OpenRaiser/Intern-Atlas huggingface dataset:
Character Transformations for Non-Autoregressive GEC Tagging
Venue: arXiv cs.CL 2021
Year: 2021

We propose a character-based nonautoregressive GEC approach, with automatically generated character transformations. Recently, per-word classification of correction edits has proven an efficient, parallelizable alternative to current encoder-decoder GEC systems. We show that word replacement edits may be suboptimal and lead to explosion of rules for spelling, diacritization and errors in morphologically rich languages, and propose a method for generating character transformations from GEC corpus. Finally, we train character transformation models for Czech, German and Russian, reaching solid results and dramatic speedup compared to autoregressive systems. The source code is released at https://github.com/ufal/wnut2021_character_transformations_gec.
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
Venue: arXiv cs.CL 2021
Year: 2021

This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107 language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform English-only pretraining when translating English speech into other languages, a setting which favors monolingual pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.
Transformation of Node to Knowledge Graph Embeddings for Faster Link Prediction in Social Networks
Venue: arXiv cs.SI 2021
Year: 2021

Recent advances in neural networks have solved common graph problems such as link prediction, node classification, node clustering, node recommendation by developing embeddings of entities and relations into vector spaces. Graph embeddings encode the structural information present in a graph. The encoded embeddings then can be used to predict the missing links in a graph. However, obtaining the optimal embeddings for a graph can be a computationally challenging task specially in an embedded system. Two techniques which we focus on in this work are 1) node embeddings from random walk based methods and 2) knowledge graph embeddings. Random walk based embeddings are computationally inexpensive to obtain but are sub-optimal whereas knowledge graph embeddings perform better but are computationally expensive. In this work, we investigate a transformation model which converts node embeddings obtained from random walk based methods to embeddings obtained from knowledge graph methods directly without an increase in the computational cost. Extensive experimentation shows that the proposed transformation model can be used for solving link prediction in real-time.
Monitoring Over the Long Term: Intermittent Deployment and Sensing Strategies for Multi-Robot Teams
Venue: arXiv cs.RO 2021
Year: 2021

In this paper, we formulate and solve the intermittent deployment problem, which yields strategies that couple \emph{when} heterogeneous robots should sense an environmental process, with where a deployed team should sense in the environment. As a motivation, suppose that a spatiotemporal process is slowly evolving and must be monitored by a multi-robot team, e.g., unmanned aerial vehicles monitoring pasturelands in a precision agriculture context. In such a case, an intermittent deployment strategy is necessary as persistent deployment or monitoring is not cost-efficient for a slowly evolving process. At the same time, the problem of where to sense once deployed must be solved as process observations yield useful feedback for determining effective future deployment and monitoring decisions. In this context, we model the environmental process to be monitored as a spatiotemporal Gaussian process with mutual information as a criterion to measure our understanding of the environment. To make the sensing resource-efficient, we demonstrate how to use matroid constraints to impose a diverse set of homogeneous and heterogeneous constraints. In addition, to reflect the cost-sensitive nature of real-world applications, we apply budgets on the cost of deployed heterogeneous robot teams. To solve the resulting problem, we exploit the theories of submodular optimization and matroids and present a greedy algorithm with bounds on sub-optimality. Finally, Monte Carlo simulations demonstrate the correctness of the proposed method.
Rethinking Drone-Based Search and Rescue with Aerial Person Detection
Venue: arXiv cs.CV 2021
Year: 2021

The visual inspection of aerial drone footage is an integral part of land search and rescue (SAR) operations today. Since this inspection is a slow, tedious and error-prone job for humans, we propose a novel deep learning algorithm to automate this aerial person detection (APD) task. We experiment with model architecture selection, online data augmentation, transfer learning, image tiling and several other techniques to improve the test performance of our method. We present the novel Aerial Inspection RetinaNet (AIR) algorithm as the combination of these contributions. The AIR detector demonstrates state-of-the-art performance on a commonly used SAR test data set in terms of both precision (~21 percentage point increase) and speed. In addition, we provide a new formal definition for the APD problem in SAR missions. That is, we propose a novel evaluation scheme that ranks detectors in terms of real-world SAR localization requirements. Finally, we propose a novel postprocessing method for robust, approximate object localization: the merging of overlapping bounding boxes (MOB) algorithm. This final processing stage used in the AIR detector significantly improves its performance and usability in the face of real-world aerial SAR missions.
Meta-Reinforcement Learning via Buffering Graph Signatures for Live Video Streaming Events
Venue: arXiv cs.NI 2021
Year: 2021

In this study, we present a meta-learning model to adapt the predictions of the network's capacity between viewers who participate in a live video streaming event. We propose the MELANIE model, where an event is formulated as a Markov Decision Process, performing meta-learning on reinforcement learning tasks. By considering a new event as a task, we design an actor-critic learning scheme to compute the optimal policy on estimating the viewers' high-bandwidth connections. To ensure fast adaptation to new connections or changes among viewers during an event, we implement a prioritized replay memory buffer based on the Kullback-Leibler divergence of the reward/throughput of the viewers' connections. Moreover, we adopt a model-agnostic meta-learning framework to generate a global model from past events. As viewers scarcely participate in several events, the challenge resides on how to account for the low structural similarity of different events. To combat this issue, we design a graph signature buffer to calculate the structural similarities of several streaming events and adjust the training of the global model accordingly. We evaluate the proposed model on the link weight prediction task on three real-world datasets of live video streaming events. Our experiments demonstrate the effectiveness of our proposed model, with an average relative gain of 25% against state-of-the-art strategies. For reproduction purposes, our evaluation datasets and implementation are publicly available at https://github.com/stefanosantaris/melanie
Mixed Dual-Hop IRS-Assisted FSO-RF Communication System with H-ARQ Protocols
Venue: arXiv cs.NI 2021
Year: 2021

Intelligent reflecting surface (IRS) is an emerging key technology for the fifth-generation (5G) and beyond wireless communication systems to provide more robust and reliable communication links. In this paper, we propose a mixed dual-hop free-space optical (FSO)-radio frequency (RF) communication system that serves the end user via a decode-and-forward (DF) relay employing hybrid automatic repeat request (HARQ) protocols on both hops. Novel closed-form expressions of the probability density function (PDF) and cumulative density function (CDF) of the equivalent end-to-end signal-to-noise ratio (SNR) are computed for the considered system. Utilizing the obtained statistics functions, we derive the outage probability (OP) and packet error rate (PER) of the proposed system by considering generalized detection techniques on the source-to-relay (S-R) link with H-ARQ protocol and IRS having phase error. We obtain useful insights into the system performance through the asymptotic analysis which aids to compute the diversity gain. The derived analytical results are validated using Monte Carlo simulation.
Impact of Weather Conditions on 5G Communication Channel under Connected Vehicles Framework
Venue: arXiv cs.NI 2021
Year: 2021

Recent research focused on improving the vehicle-to-vehicle communication (V2V) based on the 5G technology. The V2V application is important because it will reduce the risk of accidents up to 70%-80%, improve traffic management, reduce congestion, and improve fuel consumption. Autonomous vehicles applications require a high bandwidth transmission channel where the 5G communication channel would be a reliable solution to support this disruptive technology. The dedicated short-range communications (DSRC), which is characterized with a frequency bandwidth of 5.9 gigahertz (GHz) (4G spectrum), was used as vehicular connectivity with a bandwidth of up to 200 megabytes per second (mb/s) and limited capacity. The 5G band can support connected multiple autonomous vehicles with high data rates and large bandwidth. In this study, the 5G communication channel is considered as vehicular connectivity with high bandwidth in the millimeter waves spectrum range. The quality of 5G wireless communication channels between connected vehicles possibly be affected by weather conditions such as rain, snow, fog, dust, and sand. In this paper, we estimate the effect of dust and sand on the propagation of millimeter waves. The Mie model is used to investigate the effect of dust and sand storms on the propagating mm-waves. The effect of dust and sand on the communication path loss of DSRC and 5G frequency band is investigated in the case of urban freeway and rural highway settings. Results show that the attenuation of dust and sand is changed when the particle size of sand, frequency of propagating wave, and concentration of dust are changed. Finally, the new model of link margin is created to estimate the effect of dust and sand on DSCR (5.9 GHz) and 5G (28 GHz) communication path loss.
Sustainable Artificial Intelligence through Continual Learning
Venue: arXiv cs.AI 2021
Year: 2021

The increasing attention on Artificial Intelligence (AI) regulation has led to the definition of a set of ethical principles grouped into the Sustainable AI framework. In this article, we identify Continual Learning, an active area of AI research, as a promising approach towards the design of systems compliant with the Sustainable AI principles. While Sustainable AI outlines general desiderata for ethical applications, Continual Learning provides means to put such desiderata into practice.
Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence
Venue: arXiv cs.AI 2021
Year: 2021

Artificial intelligence (AI) provides a promising substitution for streamlining COVID-19 diagnoses. However, concerns surrounding security and trustworthiness impede the collection of large-scale representative medical data, posing a considerable challenge for training a well-generalised model in clinical practices. To address this, we launch the Unified CT-COVID AI Diagnostic Initiative (UCADI), where the AI model can be distributedly trained and independently executed at each host institution under a federated learning framework (FL) without data sharing. Here we show that our FL model outperformed all the local models by a large yield (test sensitivity /specificity in China: 0.973/0.951, in the UK: 0.730/0.942), achieving comparable performance with a panel of professional radiologists. We further evaluated the model on the hold-out (collected from another two hospitals leaving out the FL) and heterogeneous (acquired with contrast materials) data, provided visual explanations for decisions made by the model, and analysed the trade-offs between the model performance and the communication costs in the federated training process. Our study is based on 9,573 chest computed tomography scans (CTs) from 3,336 patients collected from 23 hospitals located in China and the UK. Collectively, our work advanced the prospects of utilising federated learning for privacy-preserving AI in digital health.
Reference-based Magnetic Resonance Image Reconstruction Using Texture Transformer
Venue: arXiv cs.CV 2021
Year: 2021

Deep Learning (DL) based methods for magnetic resonance (MR) image reconstruction have been shown to produce superior performance in recent years. However, these methods either only leverage under-sampled data or require a paired fully-sampled auxiliary modality to perform multi-modal reconstruction. Consequently, existing approaches neglect to explore attention mechanisms that can transfer textures from reference fully-sampled data to under-sampled data within a single modality, which limits these approaches in challenging cases. In this paper, we propose a novel Texture Transformer Module (TTM) for accelerated MRI reconstruction, in which we formulate the under-sampled data and reference data as queries and keys in a transformer. The TTM facilitates joint feature learning across under-sampled and reference data, so the feature correspondences can be discovered by attention and accurate texture features can be leveraged during reconstruction. Notably, the proposed TTM can be stacked on prior MRI reconstruction approaches to further improve their performance. Extensive experiments show that TTM can significantly improve the performance of several popular DL-based MRI reconstruction methods.
QGTC: Accelerating Quantized Graph Neural Networks via GPU Tensor Core
Venue: arXiv cs.DC 2021
Year: 2021

Over the most recent years, quantized graph neural network (QGNN) attracts lots of research and industry attention due to its high robustness and low computation and memory overhead. Unfortunately, the performance gains of QGNN have never been realized on modern GPU platforms. To this end, we propose the first Tensor Core (TC) based computing framework, QGTC, to support any-bitwidth computation for QGNNs on GPUs. We introduce a novel quantized low-bit arithmetic design based on the low-bit data representation and bit-decomposed computation. We craft a novel TC-tailored CUDA kernel design by incorporating 3D-stacked bit compression, zero-tile jumping, and non-zero tile reuse technique to improve the performance systematically. We incorporate an effective bandwidth-optimized subgraph packing strategy to maximize the transferring efficiency between CPU host and GPU device. We integrate QGTC with Pytorch for better programmability and extensibility. Extensive experiments demonstrate that QGTC achieves on average 2.7x speedup compared with the state-of-the-art Deep Graph Library framework across diverse settings.
Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length
Venue: arXiv cs.CL 2021
Year: 2021

Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. TinyBERT addresses the computational efficiency by self-distilling BERT into a smaller transformer representation having fewer layers and smaller internal embedding. However, TinyBERT's performance drops when we reduce the number of layers by 50%, and drops even more abruptly when we reduce the number of layers by 75% for advanced NLP tasks such as span question answering. Additionally, a separate model must be trained for each inference scenario with its distinct computational budget. In this work we present Dynamic-TinyBERT, a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss-drop). Upon publication, the code to reproduce our work will be open-sourced.
A Variant of the VC-dimension with Applications to Depth-3 Circuits
Venue: arXiv cs.CC 2021
Year: 2021

We introduce the following variant of the VC-dimension. Given $S \subseteq \{0, 1\}^n$ and a positive integer $d$, we define $\mathbb{U}_d(S)$ to be the size of the largest subset $I \subseteq [n]$ such that the projection of $S$ on every subset of $I$ of size $d$ is the $d$-dimensional cube. We show that determining the largest cardinality of a set with a given $\mathbb{U}_d$ dimension is equivalent to a Turán-type problem related to the total number of cliques in a $d$-uniform hypergraph. This allows us to beat the Sauer--Shelah lemma for this notion of dimension. We use this to obtain several results on $Σ_3^k$-circuits, i.e., depth-$3$ circuits with top gate OR and bottom fan-in at most $k$: * Tight relationship between the number of satisfying assignments of a $2$-CNF and the dimension of the largest projection accepted by it, thus improving Paturi, Saks, and Zane (Comput. Complex. '00). * Improved $Σ_3^3$-circuit lower bounds for affine dispersers for sublinear dimension. Moreover, we pose a purely hypergraph-theoretic conjecture under which we get further improvement. * We make progress towards settling the $Σ_3^2$ complexity of the inner product function and all degree-$2$ polynomials over $\mathbb{F}_2$ in general. The question of determining the $Σ_3^3$ complexity of IP was recently posed by Golovnev, Kulikov, and Williams (ITCS'21).
On the Recoverable Traveling Salesman Problem
Venue: arXiv cs.DS 2021
Year: 2021

In this paper we consider the Recoverable Traveling Salesman Problem (TSP). Here the task is to find two tours simultaneously, such that the intersection between the tours is at least a given minimum size, while the sum of travel distances with respect to two different distance metrics is minimized. Building upon the classic double-tree method, we derive a 4-approximation algorithm for the RecoverableTSP. We also show that if the required size of the intersection between the tours is constant, a 2-approximation guarantee can be achieved, even if more than two tours need to be constructed. We discuss consequences for approximability results in the more general area of recoverable robust optimization.
ClipCap: CLIP Prefix for Image Captioning
Venue: arXiv cs.CV 2021
Year: 2021

Image captioning is a fundamental task in vision-language understanding, where the model predicts a textual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. The recently proposed CLIP model contains rich semantic features which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we obtain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without additional annotations or pre-training, it efficiently generates meaningful captions for large-scale and diverse datasets. Surprisingly, our method works well even when only the mapping network is trained, while both CLIP and the language model remain frozen, allowing a lighter architecture with less trainable parameters. Through quantitative evaluation, we demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets, while it is simpler, faster, and lighter. Our code is available in https://github.com/rmokady/CLIP_prefix_caption.
Patent Sentiment Analysis to Highlight Patent Paragraphs
Venue: arXiv cs.LG 2021
Year: 2021

Given a patent document, identifying distinct semantic annotations is an interesting research aspect. Text annotation helps the patent practitioners such as examiners and patent attorneys to quickly identify the key arguments of any invention, successively providing a timely marking of a patent text. In the process of manual patent analysis, to attain better readability, recognising the semantic information by marking paragraphs is in practice. This semantic annotation process is laborious and time-consuming. To alleviate such a problem, we proposed a novel dataset to train Machine Learning algorithms to automate the highlighting process. The contributions of this work are: i) we developed a multi-class, novel dataset of size 150k samples by traversing USPTO patents over a decade, ii) articulated statistics and distributions of data using imperative exploratory data analysis, iii) baseline Machine Learning models are developed to utilize the dataset to address patent paragraph highlighting task, iv) dataset and codes relating to this task are open-sourced through a dedicated GIT web page: https://github.com/Renuk9390/Patent_Sentiment_Analysis and v) future path to extend this work using Deep Learning and domain specific pre-trained language models to develop a tool to highlight is provided. This work assist patent practitioners in highlighting semantic information automatically and aid to create a sustainable and efficient patent analysis using the aptitude of Machine Learning.
Evolutionary Algorithm for Graph Coloring Problem
Venue: arXiv cs.NE 2021
Year: 2021

The graph coloring problem (GCP) is one of the most studied NP-HARD problems in computer science. Given a graph , the task is to assign a color to all vertices such that no vertices sharing an edge receive the same color and that the number of used colors, is minimal. Different heuristic, meta-heuristic, machine learning and hybrid solution methods have been applied to obtain the solution. To solve this problem we use mutation of evolutionary algorithm. For this purpose we introduce binary encoding for Graph Coloring Problem. This binary encoding help us for mutation, evaluate, immune system and merge color easily and also reduce coloring dynamically. In the traditional evolutionary algorithm (EA) for graph coloring, k-coloring approach is used and the EA is run repeatedly until the lowest possible is reached. In our paper, we start with the theoretical upper bound of chromatic number, that is, maximum out-degree + 1 and in the process of evolution some of the colors are made unused to dynamically reduce the number of color in every generation. We test few standard DIMACS benchmark and compare resent paper. Maximum results are same as expected chromatic color and few data sets are larger than expected chromatic number