OpenRaiser/Intern-Atlas huggingface dataset:
One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency
Venue: arXiv cs.NE 2021
Year: 2021

Spiking Neural Networks (SNNs) are energy efficient alternatives to commonly used deep neural networks (DNNs). Through event-driven information processing, SNNs can reduce the expensive compute requirements of DNNs considerably, while achieving comparable performance. However, high inference latency is a significant hindrance to the edge deployment of deep SNNs. Computation over multiple timesteps not only increases latency as well as overall energy budget due to higher number of operations, but also incurs memory access overhead of fetching membrane potentials, both of which lessen the energy benefits of SNNs. To overcome this bottleneck and leverage the full potential of SNNs, we propose an Iterative Initialization and Retraining method for SNNs (IIR-SNN) to perform single shot inference in the temporal axis. The method starts with an SNN trained with T timesteps (T>1). Then at each stage of latency reduction, the network trained at previous stage with higher timestep is utilized as initialization for subsequent training with lower timestep. This acts as a compression method, as the network is gradually shrunk in the temporal domain. In this paper, we use direct input encoding and choose T=5, since as per literature, it is the minimum required latency to achieve satisfactory performance on ImageNet. The proposed scheme allows us to obtain SNNs with up to unit latency, requiring a single forward pass during inference. We achieve top-1 accuracy of 93.05%, 70.15% and 67.71% on CIFAR-10, CIFAR-100 and ImageNet, respectively using VGG16, with just 1 timestep. In addition, IIR-SNNs perform inference with 5-2500X reduced latency compared to other state-of-the-art SNNs, maintaining comparable or even better accuracy. Furthermore, in comparison with standard DNNs, the proposed IIR-SNNs provide25-33X higher energy efficiency, while being comparable to them in classification performance.
Datasets are not Enough: Challenges in Labeling Network Traffic
Venue: arXiv cs.CR 2021
Year: 2021

In contrast to previous surveys, the present work is not focused on reviewing the datasets used in the network security field. The fact is that many of the available public labeled datasets represent the network behavior just for a particular time period. Given the rate of change in malicious behavior and the serious challenge to label, and maintain these datasets, they become quickly obsolete. Therefore, this work is focused on the analysis of current labeling methodologies applied to network-based data. In the field of network security, the process of labeling a representative network traffic dataset is particularly challenging and costly since very specialized knowledge is required to classify network traces. Consequently, most of the current traffic labeling methods are based on the automatic generation of synthetic network traces, which hides many of the essential aspects necessary for a correct differentiation between normal and malicious behavior. Alternatively, a few other methods incorporate non-experts users in the labeling process of real traffic with the help of visual and statistical tools. However, after conducting an in-depth analysis, it seems that all current methods for labeling suffer from fundamental drawbacks regarding the quality, volume, and speed of the resulting dataset. This lack of consistent methods for continuously generating a representative dataset with an accurate and validated methodology must be addressed by the network security research community. Moreover, a consistent label methodology is a fundamental condition for helping in the acceptance of novel detection approaches based on statistical and machine learning techniques.
Impact of delay classes on the data structure in IOTA
Venue: arXiv cs.DC 2021
Year: 2021

In distributed ledger technologies (DLTs) with a directed acyclic graph (DAG) data structure, a message-issuing node can decide where to append that message and, consequently, how to grow the DAG. This DAG data structure can typically be decomposed into two pools of messages: referenced messages and unreferenced messages (tips). The selection of the parent messages to which a node appends the messages it issues, depends on which messages it considers as tips. However, the exact time that a message enters the tip pool of a node depends on the delay of that message. In previous works, it was considered that messages have the same or similar delay; however, this generally may not be the case. We introduce the concept of classes of delays, where messages belonging to a certain class have a specific delay, and where these classes coexist in the DAG. We provide a general model that predicts the tip pool size for any finite number of different classes. This categorisation and model is applied to the first iteration of the IOTA 2.0 protocol (a.k.a. Coordicide), where two distinct classes, namely value and data messages, coexist. We show that the tip pool size depends strongly on the dominating class that is present. Finally, we provide a methodology for controlling the tip pool size by dynamically adjusting the number of references a message creates.
Structural invariants in individuals language use: the "ego network" of words
Venue: arXiv cs.SI 2021
Year: 2021

The cognitive constraints that humans exhibit in their social interactions have been extensively studied by anthropologists, who have highlighted their regularities across different types of social networks. We postulate that similar regularities can be found in other cognitive processes, such as those involving language production. In order to provide preliminary evidence for this claim, we analyse a dataset containing tweets of a heterogeneous group of Twitter users (regular users and professional writers). Leveraging a methodology similar to the one used to uncover the well-established social cognitive constraints, we find that a concentric layered structure (which we call ego network of words, in analogy to the ego network of social relationships) very well captures how individuals organise the words they use. The size of the layers in this structure regularly grows (approximately 2-3 times with respect to the previous one) when moving outwards, and the two penultimate external layers consistently account for approximately 60% and 30% of the used words (the outermost layer contains 100% of the words), irrespective of the number of the total number of layers of the user.
Smart Crawling: A New Approach toward Focus Crawling from Twitter
Venue: arXiv cs.IR 2021
Year: 2021

Twitter is a social network that offers a rich and interesting source of information challenging to retrieve and analyze. Twitter data can be accessed using a REST API. The available operations allow retrieving tweets on the basis of a set of keywords but with limitations such as the number of calls per minute and the size of results. Besides, there is no control on retrieved results and finding tweets which are relevant to a specific topic is a big issue. Given these limitations, it is important that the query keywords cover unambiguously the topic of interest in order to both reach the relevant answers and decrease the number of API calls. In this paper, we introduce a new crawling algorithm called "SmartTwitter Crawling" (STiC) that retrieves a set of tweets related to a target topic. In this algorithm, we take an initial keyword query and enrich it using a set of additional keywords that come from different data sources. STiC algorithm relies on a DFS search in Twittergraph where each reached tweet is considered if it is relevant with the query keywords using a scoring, updated throughout the whole crawling process. This scoring takes into account the tweet text, hashtags and the users who have posted the tweet, replied to the tweet, been mentioned in the tweet or retweeted the tweet. Given this score, STiC is able to select relevant tweets in each iteration and continue by adding the related valuable tweets. Several experiments have been achieved for different kinds of queries, the results showedthat the precision increases compared to a simple BFS search.
Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Information
Venue: arXiv cs.SD 2021
Year: 2021

Automated audio captioning (AAC) has developed rapidly in recent years, involving acoustic signal processing and natural language processing to generate human-readable sentences for audio clips. The current models are generally based on the neural encoder-decoder architecture, and their decoder mainly uses acoustic information that is extracted from the CNN-based encoder. However, they have ignored semantic information that could help the AAC model to generate meaningful descriptions. This paper proposes a novel approach for automated audio captioning based on incorporating semantic and acoustic information. Specifically, our audio captioning model consists of two sub-modules. (1) The pre-trained keyword encoder utilizes pre-trained ResNet38 to initialize its parameters, and then it is trained by extracted keywords as labels. (2) The multi-modal attention decoder adopts an LSTM-based decoder that contains semantic and acoustic attention modules. Experiments demonstrate that our proposed model achieves state-of-the-art performance on the Clotho dataset. Our code can be found at https://github.com/WangHelin1997/DCASE2021_Task6_PKU
The Turing machine of a harmonic oscillator: from the code to the dynamic system
Venue: arXiv cs.OH 2021
Year: 2021

In this work we consider a dynamic system consisting of a damped harmonic oscillator and we formalize a Turing Machine whose definition in terms of states, alphabet and transition rules, can be considered equivalent to that of the oscillator. We prove that the Turing Machine of a FOR loop corresponds to that of the oscillator and we ask ourselves if it is possible to obtain the dynamic system of the harmonic oscillator as a physical realization of the FOR loop. We discuss the relationship between the results found and the science of Can and Can't. We discuss the possibility of an evolution of computer science also towards non-computerized specialized machines whose operating principle is designed as an automatic process starting from a source code instead of as a work of human ingenuity. The approach to the implementation of algorithms in dynamic systems instead of universal computers can be particularly interesting for the field of both diagnostic and implantable medical devices.
Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble
Venue: arXiv cs.CV 2021
Year: 2021

Sign language is commonly used by deaf or mute people to communicate but requires extensive effort to master. It is usually performed with the fast yet delicate movement of hand gestures, body posture, and even facial expressions. Current Sign Language Recognition (SLR) methods usually extract features via deep neural networks and suffer overfitting due to limited and noisy data. Recently, skeleton-based action recognition has attracted increasing attention due to its subject-invariant and background-invariant nature, whereas skeleton-based SLR is still under exploration due to the lack of hand annotations. Some researchers have tried to use off-line hand pose trackers to obtain hand keypoints and aid in recognizing sign language via recurrent neural networks. Nevertheless, none of them outperforms RGB-based approaches yet. To this end, we propose a novel Skeleton Aware Multi-modal Framework with a Global Ensemble Model (GEM) for isolated SLR (SAM-SLR-v2) to learn and fuse multi-modal feature representations towards a higher recognition rate. Specifically, we propose a Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics of skeleton keypoints and a Separable Spatial-Temporal Convolution Network (SSTCN) to exploit skeleton features. The skeleton-based predictions are fused with other RGB and depth based modalities by the proposed late-fusion GEM to provide global information and make a faithful SLR prediction. Experiments on three isolated SLR datasets demonstrate that our proposed SAM-SLR-v2 framework is exceedingly effective and achieves state-of-the-art performance with significant margins. Our code will be available at https://github.com/jackyjsy/SAM-SLR-v2
Offline Reinforcement Learning with Implicit Q-Learning
Venue: arXiv cs.LG 2021
Year: 2021

Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose an offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function. Then, we extract the policy via advantage-weighted behavioral cloning. We dub our method implicit Q-learning (IQL). IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.
A Brief Introduction to Automatic Differentiation for Machine Learning
Venue: arXiv cs.LG 2021
Year: 2021

Machine learning and neural network models in particular have been improving the state of the art performance on many artificial intelligence related tasks. Neural network models are typically implemented using frameworks that perform gradient based optimization methods to fit a model to a dataset. These frameworks use a technique of calculating derivatives called automatic differentiation (AD) which removes the burden of performing derivative calculations from the model designer. In this report we describe AD, its motivations, and different implementation approaches. We briefly describe dataflow programming as it relates to AD. Lastly, we present example programs that are implemented with Tensorflow and PyTorch, which are two commonly used AD frameworks.
Molecular Graph Generation via Geometric Scattering
Venue: arXiv cs.LG 2021
Year: 2021

Graph neural networks (GNNs) have been used extensively for addressing problems in drug design and discovery. Both ligand and target molecules are represented as graphs with node and edge features encoding information about atomic elements and bonds respectively. Although existing deep learning models perform remarkably well at predicting physicochemical properties and binding affinities, the generation of new molecules with optimized properties remains challenging. Inherently, most GNNs perform poorly in whole-graph representation due to the limitations of the message-passing paradigm. Furthermore, step-by-step graph generation frameworks that use reinforcement learning or other sequential processing can be slow and result in a high proportion of invalid molecules with substantial post-processing needed in order to satisfy the principles of stoichiometry. To address these issues, we propose a representation-first approach to molecular graph generation. We guide the latent representation of an autoencoder by capturing graph structure information with the geometric scattering transform and apply penalties that structure the representation also by molecular properties. We show that this highly structured latent space can be directly used for molecular graph generation by the use of a GAN. We demonstrate that our architecture learns meaningful representations of drug datasets and provides a platform for goal-directed drug synthesis.
Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation
Venue: arXiv cs.LG 2021
Year: 2021

Consistency training is a popular method to improve deep learning models in computer vision and natural language processing. Graph neural networks (GNNs) have achieved remarkable performance in a variety of network science learning tasks, but to date no work has studied the effect of consistency training on large-scale graph problems. GNNs scale to large graphs by minibatch training and subsample node neighbors to deal with high degree nodes. We utilize the randomness inherent in the subsampling of neighbors and introduce a novel consistency training method to improve accuracy. For a target node we generate different neighborhood expansions, and distill the knowledge of the average of the predictions to the GNN. Our method approximates the expected prediction of the possible neighborhood samples and practically only requires a few samples. We demonstrate that our training method outperforms standard GNN training in several different settings, and yields the largest gains when label rates are low.
NetRep: Automatic Repair for Network Programs
Venue: arXiv cs.SE 2021
Year: 2021

Debugging imperative network programs is a challenging task for developers because understanding various network modules and complicated data structures is typically time-consuming. To address the challenge, this paper presents an automated technique for repairing network programs from unit tests. Specifically, given as input a faulty network program and a set of unit tests, our approach localizes the fault through symbolic reasoning, and synthesizes a patch such that the repaired program can pass all unit tests. It applies domain-specific abstraction to simplify network data structures and utilizes modular analysis to facilitate function summary reuse for symbolic analysis. We implement the proposed techniques in a tool called NetRep and evaluate it on 10 benchmarks adapted from real-world software-defined networking controllers. The evaluation results demonstrate the effectiveness and efficiency of NetRep for repairing network programs.
Exact and Bounded Collision Probability for Motion Planning under Gaussian Uncertainty
Venue: arXiv cs.RO 2021
Year: 2021

Computing collision-free trajectories is of prime importance for safe navigation. We present an approach for computing the collision probability under Gaussian distributed motion and sensing uncertainty with the robot and static obstacle shapes approximated as ellipsoids. The collision condition is formulated as the distance between ellipsoids and unlike previous approaches we provide a method for computing the exact collision probability. Furthermore, we provide a tight upper bound that can be computed much faster during online planning. Comparison to other state-of-the-art methods is also provided. The proposed method is evaluated in simulation under varying configuration and number of obstacles.
Sub-Terahertz Spatial Statistical MIMO Channel Model for Urban Microcells at 142 GHz
Venue: arXiv cs.IT 2021
Year: 2021

Sixth generation (6G) cellular systems are expected to extend the operational range to sub-Terahertz (THz) frequencies between 100 and 300 GHz due to the broad unexploited spectrum therein. A proper channel model is needed to accurately describe spatial and temporal channel characteristics and faithfully create channel impulse responses at sub-THz frequencies. This paper studies the channel spatial statistics such as the number of spatial clusters and cluster power distribution based on recent radio propagation measurements conducted at 142 GHz in an urban microcell (UMi) scenario. For the 28 measured locations, we observe one to four spatial clusters at most locations. A detailed spatial statistical multiple input multiple output (MIMO) channel generation procedure is introduced based on the derived empirical channel statistics. We find that beamforming provides better spectral efficiency than spatial multiplexing in the LOS scenario due to the boresight path, and two spatial streams usually offer the highest spectral efficiency at most NLOS locations due to the limited number of spatial clusters.
An Overview of Ontologies and Tool Support for COVID-19 Analytics
Venue: arXiv cs.SE 2021
Year: 2021

The outbreak of the SARS-CoV-2 pandemic of the new COVID-19 disease (COVID-19 for short) demands empowering existing medical, economic, and social emergency backend systems with data analytics capabilities. An impediment in taking advantages of data analytics in these systems is the lack of a unified framework or reference model. Ontologies are highlighted as a promising solution to bridge this gap by providing a formal representation of COVID-19 concepts such as symptoms, infections rate, contact tracing, and drug modelling. Ontology-based solutions enable the integration of diverse data sources that leads to a better understanding of pandemic data, management of smart lockdowns by identifying pandemic hotspots, and knowledge-driven inference, reasoning, and recommendations to tackle surrounding issues.
Fake News Detection in Spanish Using Deep Learning Techniques
Venue: arXiv cs.CL 2021
Year: 2021

This paper addresses the problem of fake news detection in Spanish using Machine Learning techniques. It is fundamentally the same problem tackled for the English language; however, there is not a significant amount of publicly available and adequately labeled fake news in Spanish to effectively train a Machine Learning model, similarly to those proposed for the English language. Therefore, this work explores different training strategies and architectures to establish a baseline for further research in this area. Four datasets were used, two in English and two in Spanish, and four experimental schemes were tested, including a baseline with classical Machine Learning models, trained and validated using a small dataset in Spanish. The remaining schemes include state-of-the-art Deep Learning models trained (or fine-tuned) and validated in English, trained and validated in Spanish, and fitted in English and validated with automatic translated Spanish sentences. The Deep Learning architectures were built on top of different pre-trained Word Embedding representations, including GloVe, ELMo, BERT, and BETO (a BERT version trained on a large corpus in Spanish). According to the results, the best strategy was a combination of a pre-trained BETO model and a Recurrent Neural Network based on LSTM layers, yielding an accuracy of up to 80%; nonetheless, a baseline model using a Random Forest estimator obtained similar outcomes. Additionally, the translation strategy did not yield acceptable results because of the propagation error; there was also observed a significant difference in models performance when trained in English or Spanish, mainly attributable to the number of samples available for each language.