OpenRaiser/Intern-Atlas huggingface dataset:
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
Venue: arXiv cs.DC 2020
Year: 2020

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are still several tricky issues to address: improving computing efficiency while ensuring convergence, and reducing memory usage without incurring additional computing costs. We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategy of data and pipeline parallelism. We also propose a new runtime scheduling algorithm to reduce device memory usage, which is orthogonal to re-computation approach and does not come at the expense of training throughput. Experiments show that DAPPLE planner consistently outperforms strategies generated by PipeDream's planner by up to 3.23x under synchronous training scenarios, and DAPPLE runtime outperforms GPipe by 1.6x speedup of training throughput and reduces the memory consumption of 12% at the same time.
Scene Graph Reasoning for Visual Question Answering
Venue: arXiv cs.LG 2020
Year: 2020

Visual question answering is concerned with answering free-form questions about an image. Since it requires a deep linguistic understanding of the question and the ability to associate it with various objects that are present in the image, it is an ambitious task and requires techniques from both computer vision and natural language processing. We propose a novel method that approaches the task by performing context-driven, sequential reasoning based on the objects and their semantic and spatial relationships present in the scene. As a first step, we derive a scene graph which describes the objects in the image, as well as their attributes and their mutual relationships. A reinforcement agent then learns to autonomously navigate over the extracted scene graph to generate paths, which are then the basis for deriving answers. We conduct a first experimental study on the challenging GQA dataset with manually curated scene graphs, where our method almost reaches the level of human performance.
The use of personal data in French public services: e-mails, websites, apps
Venue: arXiv cs.CY 2020
Year: 2020

The study we carried out enabled us to extract some conclusions, which are contrasted with the results obtained. First, in the field of mobile applications, permissions and tracers are almost always present. Android, as far as PlayStore permissions are concerned, is the main entity concerning this domain. Under the pretext of guaranteeing an optimal functioning of the applications, these permissions can sometimes hide some very dangerous tracing means for the users. Google, a major player in tracing and a major power in the storage of information of net users, is behind most tracers. Trackers have two main missions. On the one hand, they allow the application to work, like Facebook's trackers that are used to log into the application or Google's trackers that allow either to trace crashes or to analyze how the application is used. On the other hand, they allow you to manage the advertising that appears in the application, which can be targeted or not. Regarding tracking in emails, we find stakeholders quite present: Google, Xiti and Iroquois. Even if they are most often used in the context of hearing measurements, they are present in public service emails. Finally with websites, Google is very present in government websites. We find common actors for applications and emails such as GAFAM or Xiti. Most of the cookies present are for audience measurement and advertising display which is often targeted.
Modelling heterogeneous outcomes in multi-agent systems
Venue: arXiv cs.MA 2020
Year: 2020

A broad set of empirical phenomenon in the study of social, economic and machine behaviour can be modelled as complex systems with averaging dynamics. However many of these models naturally result in consensus or consensus-like outcomes. In reality, empirical phenomenon rarely converge to these and instead are characterized by rich, persistent variation in the agent states. Such heterogeneous outcomes are a natural consequence of a number of models that incorporate external perturbation to the otherwise convex dynamics of the agents. The purpose of this paper is to formalize the notion of heterogeneity and demonstrate which classes of models are able to achieve it as an outcome, and therefore are better suited to modelling important empirical questions. We do so by determining how the topology of (time-varying) interaction networks restrict the space of possible steady-state outcomes for agents, and how this is related to the study of random walks on graphs. We consider a number of intentionally diverse examples to demonstrate how the results can be applied.
Frequency-Mixing Intelligent Reflecting Surfaces for Nonlinear Wireless Propagation
Venue: arXiv cs.IT 2020
Year: 2020

We introduce the concept of frequency-mixing intelligent reflecting surface (FMx-IRS), where the elements of the surface continuously change the phases of the incident signals. In this way, the FMx-IRS acts as a frequency mixer and makes the propagation environment nonlinear, thereby introducing new frequencies. We study the basic features of the proposed concept and demonstrate its advantages that stem from the novel type of control over the wireless propagation. The channel decoupling feature and the correlation between reflected channels are elaborated for the architecture, and are validated by the simulations.
PrototypeML: A Neural Network Integrated Design and Development Environment
Venue: arXiv cs.LG 2020
Year: 2020

Neural network architectures are most often conceptually designed and described in visual terms, but are implemented by writing error-prone code. PrototypeML is a machine learning development environment that bridges the dichotomy between the design and development processes: it provides a highly intuitive visual neural network design interface that supports (yet abstracts) the full capabilities of the PyTorch deep learning framework, reduces model design and development time, makes debugging easier, and automates many framework and code writing idiosyncrasies. In this paper, we detail the deep learning development deficiencies that drove the implementation of PrototypeML, and propose a hybrid approach to resolve these issues without limiting network expressiveness or reducing code quality. We demonstrate the real-world benefits of a visual approach to neural network design for research, industry and teaching. Available at https://PrototypeML.com
MICN: a network coding protocol for ICN with multiple distinct interests per generation
Venue: arXiv cs.NI 2020
Year: 2020

In Information-Centric Networking (ICN), consumers send interest packets to the network and receive data packets as a response to their request without taking care of the producers, which have provided the content, contrary to conventional IP networks. ICN supports the use of multiple paths; however, with multiple consumers and producers, coordination among the nodes is required to efficiently use the network resources. Network coding (NC) is a promising tool to address this issue. The challenge in the case of NC is to be able to get independent coded content in response to multiple parallel interests by one or several consumers. In this work, we propose a novel construction called MILIC (Multiple Interests for Linearly Independent Contents) that impose constraints on how the replies to interests are coded, intending to get linearly independent contents in response to multiple interests. Several protocol variants, called MICN (MILIC-ICN), built on top of NDN (Named Data Networking), are proposed to integrate these interest constraints and NC of data packets. Numerical analysis and simulations illustrate that the MILIC construction performs well and that the MICN protocols are close to optimal throughput on some scenarios. MICN protocols compare favorably to existing protocols and show significant benefits when considering the total number of transmitted packets in the network, and in the case of high link loss rate.
Federated Learning with Compression: Unified Analysis and Sharp Guarantees
Venue: arXiv cs.LG 2020
Year: 2020

In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices with potentially unreliable or limited communication and heterogeneous data distributions. Two notable trends to deal with the communication overhead of federated algorithms are gradient compression and local computation with periodic communication. Despite many attempts, characterizing the relationship between these two approaches has proven elusive. We address this by proposing a set of algorithms with periodical compressed (quantized or sparsified) communication and analyze their convergence properties in both homogeneous and heterogeneous local data distribution settings. For the homogeneous setting, our analysis improves existing bounds by providing tighter convergence rates for both strongly convex and non-convex objective functions. To mitigate data heterogeneity, we introduce a local gradient tracking scheme and obtain sharp convergence rates that match the best-known communication complexities without compression for convex, strongly convex, and nonconvex settings. We complement our theoretical results and demonstrate the effectiveness of our proposed methods by several experiments on real-world datasets.
Data-Driven Method for Enhanced Corrosion Assessment of Reinforced Concrete Structures
Venue: arXiv cs.LG 2020
Year: 2020

Corrosion is a major problem affecting the durability of reinforced concrete structures. Corrosion related maintenance and repair of reinforced concrete structures cost multibillion USD per annum globally. It is often triggered by the ingression of carbon dioxide and/or chloride into the pores of concrete. Estimation of these corrosion causing factors using the conventional models results in suboptimal assessment since they are incapable of capturing the complex interaction of parameters. Hygrothermal interaction also plays a role in aggravating the corrosion of reinforcement bar and this is usually counteracted by applying surface-protection systems. These systems have different degree of protection and they may even cause deterioration to the structure unintentionally. The overall objective of this dissertation is to provide a framework that enhances the assessment reliability of the corrosion controlling factors. The framework is realized through the development of data-driven carbonation depth, chloride profile and hygrothermal performance prediction models. The carbonation depth prediction model integrates neural network, decision tree, boosted and bagged ensemble decision trees. The ensemble tree based chloride profile prediction models evaluate the significance of chloride ingress controlling variables from various perspectives. The hygrothermal interaction prediction models are developed using neural networks to evaluate the status of corrosion and other unexpected deteriorations in surface-treated concrete elements. Long-term data for all models were obtained from three different field experiments. The performance comparison of the developed carbonation depth prediction model with the conventional one confirmed the prediction superiority of the data-driven model. The variable ...
Reinforcement Learning Based Handwritten Digit Recognition with Two-State Q-Learning
Venue: arXiv cs.CV 2020
Year: 2020

We present a simple yet efficient Hybrid Classifier based on Deep Learning and Reinforcement Learning. Q-Learning is used with two Q-states and four actions. Conventional techniques use feature maps extracted from Convolutional Neural Networks (CNNs) and include them in the Qstates along with past history. This leads to difficulties with these approaches as the number of states is very large number due to high dimensions of the feature maps. Since our method uses only two Q-states it is simple and has much lesser number of parameters to optimize and also thus has a straightforward reward function. Also, the approach uses unexplored actions for image processing vis-a-vis other contemporary techniques. Three datasets have been used for benchmarking of the approach. These are the MNIST Digit Image Dataset, the USPS Digit Image Dataset and the MATLAB Digit Image Dataset. The performance of the proposed hybrid classifier has been compared with other contemporary techniques like a well-established Reinforcement Learning Technique, AlexNet, CNN-Nearest Neighbor Classifier and CNNSupport Vector Machine Classifier. Our approach outperforms these contemporary hybrid classifiers on all the three datasets used.
Parallel Planar Subgraph Isomorphism and Vertex Connectivity
Venue: arXiv cs.DS 2020
Year: 2020

We present the first parallel fixed-parameter algorithm for subgraph isomorphism in planar graphs, bounded-genus graphs, and, more generally, all minor-closed graphs of locally bounded treewidth. Our randomized low depth algorithm has a near-linear work dependency on the size of the target graph. Existing low depth algorithms do not guarantee that the work remains asymptotically the same for any constant-sized pattern. By using a connection to certain separating cycles, our subgraph isomorphism algorithm can decide the vertex connectivity of a planar graph (with high probability) in asymptotically near-linear work and poly-logarithmic depth. Previously, no sub-quadratic work and poly-logarithmic depth bound was known in planar graphs (in particular for distinguishing between four-connected and five-connected planar graphs).
Exponentially Weighted l_2 Regularization Strategy in Constructing Reinforced Second-order Fuzzy Rule-based Model
Venue: arXiv cs.LG 2020
Year: 2020

In the conventional Takagi-Sugeno-Kang (TSK)-type fuzzy models, constant or linear functions are usually utilized as the consequent parts of the fuzzy rules, but they cannot effectively describe the behavior within local regions defined by the antecedent parts. In this article, a theoretical and practical design methodology is developed to address this problem. First, the information granulation (Fuzzy C-Means) method is applied to capture the structure in the data and split the input space into subspaces, as well as form the antecedent parts. Second, the quadratic polynomials (QPs) are employed as the consequent parts. Compared with constant and linear functions, QPs can describe the input-output behavior within the local regions (subspaces) by refining the relationship between input and output variables. However, although QP can improve the approximation ability of the model, it could lead to the deterioration of the prediction ability of the model (e.g., overfitting). To handle this issue, we introduce an exponential weight approach inspired by the weight function theory encountered in harmonic analysis. More specifically, we adopt the exponential functions as the targeted penalty terms, which are equipped with l2 regularization (l2) (i.e., exponential weighted l2, ewl_2) to match the proposed reinforced second-order fuzzy rule-based model (RSFRM) properly. The advantage of el 2 compared to ordinary l2 lies in separately identifying and penalizing different types of polynomial terms in the coefficient estimation, and its results not only alleviate the overfitting and prevent the deterioration of generalization ability but also effectively release the prediction potential of the model.
Localization and Mapping of Sparse Geologic Features with Unpiloted Aircraft Systems
Venue: arXiv cs.RO 2020
Year: 2020

Robotic mapping is attractive in many scientific applications that involve environmental surveys. This paper presents a system for localization and mapping of sparsely distributed surface features such as precariously balanced rocks (PBRs), whose geometric fragility parameters provide valuable information on earthquake processes and landscape development. With this geomorphologic problem as the test domain, we carry out a lawn-mower search pattern from a high elevation using an Unpiloted Aerial Vehicle (UAV) equipped with a flight controller, GPS module, stereo camera, and onboard computer. Once a potential PBR target is detected by a deep neural network in real time, we track its bounding box in the image coordinates by applying a Kalman filter that fuses the deep learning detection with Kanade-Lucas-Tomasi (KLT) tracking. The target is localized in world coordinates using depth filtering where a set of 3D points are filtered by object bounding boxes from different camera perspectives. The 3D points also provide a strong prior on target shape, which is used for UAV path planning to closely map the target using RGBD SLAM. After target mapping, the UAS resumes the lawn-mower search pattern to locate and map the next target.
Verifiably Safe Exploration for End-to-End Reinforcement Learning
Venue: arXiv cs.AI 2020
Year: 2020

Deploying deep reinforcement learning in safety-critical settings requires developing algorithms that obey hard constraints during exploration. This paper contributes a first approach toward enforcing formal safety constraints on end-to-end policies with visual inputs. Our approach draws on recent advances in object detection and automated reasoning for hybrid dynamical systems. The approach is evaluated on a novel benchmark that emphasizes the challenge of safely exploring in the presence of hard constraints. Our benchmark draws from several proposed problem sets for safe learning and includes problems that emphasize challenges such as reward signals that are not aligned with safety constraints. On each of these benchmark problems, our algorithm completely avoids unsafe behavior while remaining competitive at optimizing for as much reward as is safe. We also prove that our method of enforcing the safety constraints preserves all safe policies from the original environment.
Climbing the WOL: Training for Cheaper Inference
Venue: arXiv cs.IR 2020
Year: 2020

Efficient inference for wide output layers (WOLs) is an essential yet challenging task in large scale machine learning. Most approaches reduce this problem to approximate maximum inner product search (MIPS), which relies heavily on the observation that for a given model, ground truth labels correspond to logits of highest value during full model inference. However, such an assumption is restrictive in practice. In this paper, we argue that approximate MIPS subroutines, despite having sub-linear computation time, are sub-optimal because they are tailored for retrieving large inner products with high recall instead of retrieving the correct labels. With WOL, the labels often have moderate inner products, which makes approximate MIPS more challenging. We propose an alternative problem formulation, called Label Superior Sampling (LSS), where the objective is to tailor the system to ensure retrieval of the correct label. Accordingly, we propose a novel learned hash approach, which is significantly more efficient and sufficient for high inference accuracy than MIPS baselines. Our extensive evaluation indicates that LSS can match or even outperform full inference accuracy with around 5x speed up and 87% energy reduction.
A Perspective on Gaussian Processes for Earth Observation
Venue: arXiv cs.LG 2020
Year: 2020

Earth observation (EO) by airborne and satellite remote sensing and in-situ observations play a fundamental role in monitoring our planet. In the last decade, machine learning and Gaussian processes (GPs) in particular has attained outstanding results in the estimation of bio-geo-physical variables from the acquired images at local and global scales in a time-resolved manner. GPs provide not only accurate estimates but also principled uncertainty estimates for the predictions, can easily accommodate multimodal data coming from different sensors and from multitemporal acquisitions, allow the introduction of physical knowledge, and a formal treatment of uncertainty quantification and error propagation. Despite great advances in forward and inverse modelling, GP models still have to face important challenges that are revised in this perspective paper. GP models should evolve towards data-driven physics-aware models that respect signal characteristics, be consistent with elementary laws of physics, and move from pure regression to observational causal inference.
Autonomous and cooperative design of the monitor positions for a team of UAVs to maximize the quantity and quality of detected objects
Venue: arXiv cs.RO 2020
Year: 2020

This paper tackles the problem of positioning a swarm of UAVs inside a completely unknown terrain, having as objective to maximize the overall situational awareness. The situational awareness is expressed by the number and quality of unique objects of interest, inside the UAVs' fields of view. YOLOv3 and a system to identify duplicate objects of interest were employed to assign a single score to each UAVs' configuration. Then, a novel navigation algorithm, capable of optimizing the previously defined score, without taking into consideration the dynamics of either UAVs or environment, is proposed. A cornerstone of the proposed approach is that it shares the same convergence characteristics as the block coordinate descent (BCD) family of approaches. The effectiveness and performance of the proposed navigation scheme were evaluated utilizing a series of experiments inside the AirSim simulator. The experimental evaluation indicates that the proposed navigation algorithm was able to consistently navigate the swarm of UAVs to "strategic" monitoring positions and also adapt to the different number of swarm sizes. Source code is available at https://github.com/dimikout3/ConvCAOAirSim.
AutoBayes: Automated Bayesian Graph Exploration for Nuisance-Robust Inference
Venue: arXiv cs.LG 2020
Year: 2020

Learning data representations that capture task-related features, but are invariant to nuisance variations remains a key challenge in machine learning. We introduce an automated Bayesian inference framework, called AutoBayes, that explores different graphical models linking classifier, encoder, decoder, estimator and adversarial network blocks to optimize nuisance-invariant machine learning pipelines. AutoBayes also enables learning disentangled representations, where the latent variable is split into multiple pieces to impose various relationships with the nuisance variation and task labels. We benchmark the framework on several public datasets, and provide analysis of its capability for subject-transfer learning with/without variational modeling and adversarial training. We demonstrate a significant performance improvement with ensemble learning across explored graphical models.