OpenRaiser/Intern-Atlas huggingface dataset:
ImageNet-21K Pretraining for the Masses
Venue: arXiv cs.CV 2021
Year: 2021

ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K
Personalizing Performance Regression Models to Black-Box Optimization Problems
Venue: arXiv cs.NE 2021
Year: 2021

Accurately predicting the performance of different optimization algorithms for previously unseen problem instances is crucial for high-performing algorithm selection and configuration techniques. In the context of numerical optimization, supervised regression approaches built on top of exploratory landscape analysis are becoming very popular. From the point of view of Machine Learning (ML), however, the approaches are often rather naive, using default regression or classification techniques without proper investigation of the suitability of the ML tools. With this work, we bring to the attention of our community the possibility to personalize regression models to specific types of optimization problems. Instead of aiming for a single model that works well across a whole set of possibly diverse problems, our personalized regression approach acknowledges that different models may suite different types of problems. Going one step further, we also investigate the impact of selecting not a single regression model per problem, but personalized ensembles. We test our approach on predicting the performance of numerical optimization heuristics on the BBOB benchmark collection.
Network Space Search for Pareto-Efficient Spaces
Venue: arXiv cs.CV 2021
Year: 2021

Network spaces have been known as a critical factor in both handcrafted network designs or defining search spaces for Neural Architecture Search (NAS). However, an effective space involves tremendous prior knowledge and/or manual effort, and additional constraints are required to discover efficiency-aware architectures. In this paper, we define a new problem, Network Space Search (NSS), as searching for favorable network spaces instead of a single architecture. We propose an NSS method to directly search for efficient-aware network spaces automatically, reducing the manual effort and immense cost in discovering satisfactory ones. The resultant network spaces, named Elite Spaces, are discovered from Expanded Search Space with minimal human expertise imposed. The Pareto-efficient Elite Spaces are aligned with the Pareto front under various complexity constraints and can be further served as NAS search spaces, benefiting differentiable NAS approaches (e.g. In CIFAR-100, an averagely 2.3% lower error rate and 3.7% closer to target constraint than the baseline with around 90% fewer samples required to find satisfactory networks). Moreover, our NSS approach is capable of searching for superior spaces in future unexplored spaces, revealing great potential in searching for network spaces automatically. Website: https://minhungchen.netlify.app/publication/nss.
TeraMIMO: A Channel Simulator for Wideband Ultra-Massive MIMO Terahertz Communications
Venue: arXiv cs.IT 2021
Year: 2021

Following recent advances in terahertz (THz) technology, there is a consensus on the crucial role of THz communications in the next generation of wireless systems. Aiming at catalyzing THz communications research, we propose TeraMIMO, an accurate stochastic MATLAB simulator of statistical THz channels. We simulate ultra-massive multiple-input multiple-output antenna configurations as critical infrastructure enablers that overcome the limitation in THz communications distances. We consider both line-of-sight and multipath components and propose frequency- and delay-domain implementations for single- and multi-carrier paradigms in both time-invariant and time-variant scenarios. We implement exhaustive molecular absorption computations based on radiative transfer theory alongside alternative sub-THz approximations. We further model THz-specific constraints, including wideband beam split effects, spherical wave propagation, and misalignment fading. We verify TeraMIMO by analogy with measurement-based channel models in the literature and ergodic capacity analysis. We introduce a graphical user interface and a guide for using TeraMIMO in THz channel generation and analyses.
Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition
Venue: arXiv cs.CV 2021
Year: 2021

In the real world, medical datasets often exhibit a long-tailed data distribution (i.e., a few classes occupy most of the data, while most classes have rarely few samples), which results in a challenging imbalance learning scenario. For example, there are estimated more than 40 different kinds of retinal diseases with variable morbidity, however with more than 30+ conditions are very rare from the global patient cohorts, which results in a typical long-tailed learning problem for deep learning-based screening models. In this study, we propose class subset learning by dividing the long-tailed data into multiple class subsets according to prior knowledge, such as regions and phenotype information. It enforces the model to focus on learning the subset-specific knowledge. More specifically, there are some relational classes that reside in the fixed retinal regions, or some common pathological features are observed in both the majority and minority conditions. With those subsets learnt teacher models, then we are able to distill the multiple teacher models into a unified model with weighted knowledge distillation loss. The proposed framework proved to be effective for the long-tailed retinal diseases recognition task. The experimental results on two different datasets demonstrate that our method is flexible and can be easily plugged into many other state-of-the-art techniques with significant improvements.
MRRT: Multiple Rapidly-Exploring Random Trees for Fast Online Replanning in Dynamic Environments
Venue: arXiv cs.RO 2021
Year: 2021

This paper presents a novel algorithm, called MRRT, which uses multiple rapidly-exploring random trees for fast online replanning of autonomous vehicles in dynamic environments with moving obstacles. The proposed algorithm is built upon the RRT algorithm with a multi-tree structure. At the beginning, the RRT algorithm is applied to find the initial solution based on partial knowledge of the environment. Then, the robot starts to execute this path. At each iteration, the new obstacle configurations are collected by the robot's sensor and used to replan the path. This new information can come from unknown static obstacles (e.g., seafloor layout) as well as moving obstacles. Then, to accommodate the environmental changes, two procedures are adopted: 1) edge pruning, and 2) tree regrowing. Specifically, the edge pruning procedure checks the collision status through the tree and only removes the invalid edges while maintaining the tree structure of already-explored regions. Due to removal of invalid edges, the tree could be broken into multiple disjoint trees. As such, the RRT algorithm is applied to regrow the trees. Specifically, a sample is created randomly and joined to all the disjoint trees in its local neighborhood by connecting to the nearest nodes. Finally, a new solution is found for the robot. The advantages of the proposed MRRT algorithm are as follows: i) retains the maximal tree structure by only pruning the edges which collide with the obstacles, ii) guarantees probabilistic completeness, and iii) is computational efficient for fast replanning since all disjoint trees are maintained for future connections and expanded simultaneously.
Open and Programmable 5G Network-in-a-Box: Technology Demonstration and Evaluation Results
Venue: arXiv cs.NI 2021
Year: 2021

The fifth-generation (5G) mobile/cellular technology is a game changer for industrial systems. Private 5G deployments are promising to address the challenges faced by industrial networks. Programmability and open-source are two key aspects which bring unprecedented flexibility and customizability to private 5G networks. Recent regulatory initiatives are removing barriers for industrial stakeholders to deploy their own local 5G networks with dedicated equipment. To this end, this demonstration showcases an open and programmable 5G network-in-a-box solution for private deployments. The network-in-a-box provides an integrated solution, based on open-source software stack and general-purpose hardware, for operation in 5G non-standalone (NSA) as well as 4G long-term evolution (LTE) modes. The demonstration also shows the capability of operation in different sub-6 GHz frequency bands, some of which are specifically available for private networks. Performance results, in terms of end-to-end latency and data rates, with a commercial off-the-shelf (COTS) 5G device are shown as well.
On Doped SC-LDPC Codes for Streaming
Venue: arXiv cs.IT 2021
Year: 2021

In streaming applications, doping improves the performance of spatially-coupled low-density parity-check (SC-LDPC) codes by creating reduced-degree check nodes in the coupled chain. We formulate a scaling law to predict the bit and block error rate of periodically-doped semi-infinite SC-LDPC code ensembles streamed over the binary erasure channel under sliding window decoding for a given finite component block length. The scaling law assumes that with some probability doping is equivalent to full termination and triggers two decoding waves; otherwise, decoding performs as if the coupled chain had not been doped at all. We approximate that probability and use the derived scaling laws to predict the error rates of SC-LDPC code ensembles in the presence of doping. The proposed scaling law provides accurate error rate predictions. We further use it to show that in streaming applications periodic doping can yield higher rates than periodic full termination for the same error-correcting performance.
Synchronization of Tree Parity Machines using non-binary input vectors
Venue: arXiv cs.CR 2021
Year: 2021

Neural cryptography is the application of artificial neural networks in the subject of cryptography. The functionality of this solution is based on a tree parity machine. It uses artificial neural networks to perform secure key exchange between network entities. This article proposes improvements to the synchronization of two tree parity machines. The improvement is based on learning artificial neural network using input vectors which have a wider range of values than binary ones. As a result, the duration of the synchronization process is reduced. Therefore, tree parity machines achieve common weights in a shorter time due to the reduction of necessary bit exchanges. This approach improves the security of neural cryptography
On the fixed-parameter tractability of the partial vertex cover problem with a matching constraint in edge-weighted bipartite graphs
Venue: arXiv cs.DM 2021
Year: 2021

In the classical partial vertex cover problem, we are given a graph $G$ and two positive integers $R$ and $L$. The goal is to check whether there is a subset $V'$ of $V$ of size at most $R$, such that $V'$ covers at least $L$ edges of $G$. The problem is NP-hard as it includes the Vertex Cover problem. Previous research has addressed the extension of this problem where one has weight-functions defined on sets of vertices and edges of $G$. In this paper, we consider the following version of the problem where on the input we are given an edge-weighted bipartite graph $G$, and three positive integers $R$, $S$ and $T$. The goal is to check whether $G$ has a subset $V'$ of vertices of $G$ of size at most $R$, such that the edges of $G$ covered by $V'$ have weight at least $S$ and they include a matching of weight at least $T$. In the paper, we address this problem from the perspective of fixed-parameter tractability. One of our hardness results is obtained via a reduction from the bi-objective knapsack problem, which we show to be W[1]-hard with respect to one of parameters. We believe that this problem might be useful in obtaining similar results in other situations.
Literature review on vulnerability detection using NLP technology
Venue: arXiv cs.CR 2021
Year: 2021

Vulnerability detection has always been the most important task in the field of software security. With the development of technology, in the face of massive source code, automated analysis and detection of vulnerabilities has become a current research hotspot. For special text files such as source code, using some of the hottest NLP technologies to build models and realize the automatic analysis and detection of source code has become one of the most anticipated studies in the field of vulnerability detection. This article does a brief survey of some recent new documents and technologies, such as CodeBERT, and summarizes the previous technologies.
Equivariant Wavelets: Fast Rotation and Translation Invariant Wavelet Scattering Transforms
Venue: arXiv cs.CV 2021
Year: 2021

Wavelet scattering networks, which are convolutional neural networks (CNNs) with fixed filters and weights, are promising tools for image analysis. Imposing symmetry on image statistics can improve human interpretability, aid in generalization, and provide dimension reduction. In this work, we introduce a fast-to-compute, translationally invariant and rotationally equivariant wavelet scattering network (EqWS) and filter bank of wavelets (triglets). We demonstrate the interpretability and quantify the invariance/equivariance of the coefficients, briefly commenting on difficulties with implementing scale equivariance. On MNIST, we show that training on a rotationally invariant reduction of the coefficients maintains rotational invariance when generalized to test data and visualize residual symmetry breaking terms. Rotation equivariance is leveraged to estimate the rotation angle of digits and reconstruct the full rotation dependence of each coefficient from a single angle. We benchmark EqWS with linear classifiers on EMNIST and CIFAR-10/100, introducing a new second-order, cross-color channel coupling for the color images. We conclude by comparing the performance of an isotropic reduction of the scattering coefficients and RWST, a previous coefficient reduction, on an isotropic classification of magnetohydrodynamic simulations with astrophysical relevance.
Real-Time Trajectory Optimization in Robot-Assisted Exercise and Rehabilitation
Venue: arXiv cs.RO 2021
Year: 2021

This work focuses on the optimization of the training trajectory orientation using a robot as an advanced exercise machine (AEM) and muscle activations as biofeedback. Muscle recruitment patterns depend on trajectory parameters of the AEMs and correlate with the efficiency of exercise. Thus, improvements to training efficiency may be achieved by optimizing these parameters. The optimal regulation of these parameters is challenging because of the complexity of the physiological dynamics from person to person as a result of the unique physical features such as musculoskeletal distribution. Furthermore, these effects can vary due to fatigue, body temperature, and other physiological factors. In this paper, a model-free optimization method using Extremum Seeking Control (ESC) as a real-time optimizer is proposed. After selecting a muscle objective, this method seeks for the optimal combination of parameters using the muscle activations as biofeedback. The muscle objective can be selected by a therapist to emphasize or de-emphasize certain muscle groups. The feasibility of this method has been proven for the automatic regulation of an ellipsoidal curve orientation, suggesting the existence of two local optimal orientations. This methodology can also be applied to other parameter regulations using a different physiological effects such as oxygen consumption and heart rate as biofeedback.
Low Complex, Narrowband-Interference Robust Synchronization for NC-OFDM Cognitive Radio
Venue: arXiv cs.SI 2021
Year: 2021

This paper presents a new, low-complexity algorithm for time and frequency synchronization in a non-contiguous-orthogonal frequency division multiplexing (NC-OFDM) radio communication system in the presence of narrowband interference (NBI). This interference scenario is plausible for cognitive radio systems. The computational complexity of the proposed synchronization method is similar to the known Schmidl&Cox algorithm. The algorithm is partially blind, i.e., it does not require information on the NC-OFDM subcarriers used, nor on the center-frequency of the NBI. Moreover, it does not require filtering to remove the NBI. It performs best for the NBI of constant carrier frequency, but it also provides good results for the frequency-modulated interfering signal.
A QUBO Algorithm to Compute Eigenvectors of Symmetric Matrices
Venue: arXiv cs.ET 2021
Year: 2021

We describe an algorithm to compute the extremal eigenvalues and corresponding eigenvectors of a symmetric matrix by solving a sequence of Quadratic Binary Optimization problems. This algorithm is robust across many different classes of symmetric matrices, can compute the eigenvector/eigenvalue pair to essentially arbitrary precision, and with minor modifications can also solve the generalized eigenvalue problem. Performance is analyzed on small random matrices and selected larger matrices from practical applications.
Federated Double Deep Q-learning for Joint Delay and Energy Minimization in IoT networks
Venue: arXiv cs.NI 2021
Year: 2021

In this paper, we propose a federated deep reinforcement learning framework to solve a multi-objective optimization problem, where we consider minimizing the expected long-term task completion delay and energy consumption of IoT devices. This is done by optimizing offloading decisions, computation resource allocation, and transmit power allocation. Since the formulated problem is a mixed-integer non-linear programming (MINLP), we first cast our problem as a multi-agent distributed deep reinforcement learning (DRL) problem and address it using double deep Q-network (DDQN), where the actions are offloading decisions. The immediate cost of each agent is calculated through solving either the transmit power optimization or local computation resource optimization, based on the selected offloading decisions (actions). Then, to enhance the learning speed of IoT devices (agents), we incorporate federated learning (FDL) at the end of each episode. FDL enhances the scalability of the proposed DRL framework, creates a context for cooperation between agents, and minimizes their privacy concerns. Our numerical results demonstrate the efficacy of our proposed federated DDQN framework in terms of learning speed compared to federated deep Q network (DQN) and non-federated DDQN algorithms. In addition, we investigate the impact of batch size, network layers, DDQN target network update frequency on the learning speed of the FDL.
Infant Vocal Tract Development Analysis and Diagnosis by Cry Signals with CNN Age Classification
Venue: arXiv cs.SD 2021
Year: 2021

From crying to babbling and then to speech, infant's vocal tract goes through anatomic restructuring. In this paper, we propose a non-invasive fast method of using infant cry signals with convolutional neural network (CNN) based age classification to diagnose the abnormality of the vocal tract development as early as 4-month age. We study F0, F1, F2, and spectrograms and relate them to the postnatal development of infant vocalization. A novel CNN based age classification is performed with binary age pairs to discover the pattern and tendency of the vocal tract changes. The effectiveness of this approach is evaluated on Baby2020 with healthy infant cries and Baby Chillanto database with pathological infant cries. The results show that our approach yields 79.20% accuracy for healthy cries, 84.80% for asphyxiated cries, and 91.20% for deaf cries. Our method first reveals that infants' vocal tract develops to a certain level at 4-month age and infants can start controlling the vocal folds to produce discontinuous cry sounds leading to babbling. Early diagnosis of growth abnormality of the vocal tract can help parents keep vigilant and adopt medical treatment or training therapy for their infants as early as possible.
Accuracy and precision: a new view on kinematic assessment of solid-state hinges and compliant mechanisms
Venue: arXiv cs.RO 2021
Year: 2021

Compliant mechanisms are alternatives to conventional mechanisms which exploit elastic strain to produce desired deformations instead of using moveable parts. They are designed for a kinematic task (providing desired deformations) but do not possess a kinematics in the strict sense. This leads to difficulties while assessing the quality of a compliant mechanism's design. The kinematics of a compliant mechanism can be seen as a fuzzy property. There is no unique kinematics, since every deformation need a particular force system to act; however, certain deformations are easier to obtain than others. A parallel can be made with measurement theory: the measured value of a quantity is not unique, but exists as statistic distribution of measures. A representative measure of this distribution can be chosen to evaluate how far the measures divert from a reference value. Based on this analogy, the concept of accuracy and precision of compliant systems are introduced and discussed in this paper. A quantitative determination of these qualities based on the eigenvalue analysis of the hinge's stiffness is proposed. This new approach is capable of removing most of the ambiguities included in the state-of-the-art assessment criteria (usually based on the concepts of path deviation and parasitic motion).
SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
Venue: arXiv cs.CV 2021
Year: 2021

Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding in a data-driven multi-task manner. To enable robust capture under complex motion patterns, we propose an effective motion embedding module to recover both the implicit motion embedding and explicit 3D motion details via a corresponding mapping function as well as a sub-motion classifier. Based on such hybrid motion information, we introduce a multi-stream spatial-temporal Graph Convolutional Network(ST-GCN) to predict the fine-grained semantic action attributes, and adopt a semantic attribute mapping block to assemble various correlated action attributes into a high-level action label for the overall detailed understanding of the whole sequence, so as to enable various applications like action assessment or motion scoring. Comprehensive experiments on both public and our proposed datasets show that with a challenging monocular sports video input, our novel approach not only significantly improves the accuracy of 3D human motion capture, but also recovers accurate fine-grained semantic action attributes.