OpenRaiser/Intern-Atlas huggingface dataset:
Adaptive Transformers for Learning Multimodal Representations
Venue: arXiv cs.CL 2020
Year: 2020

The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. In this work, we extend adaptive approaches to learn more about model interpretability and computational efficiency. Specifically, we study attention spans, sparse, and structured dropout methods to help understand how their attention mechanism extends for vision and language tasks. We further show that these approaches can help us learn more about how the network perceives the complexity of input sequences, sparsity preferences for different modalities, and other related phenomena.
COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter
Venue: arXiv cs.CL 2020
Year: 2020

In this work, we release COVID-Twitter-BERT (CT-BERT), a transformer-based model, pretrained on a large corpus of Twitter messages on the topic of COVID-19. Our model shows a 10-30% marginal improvement compared to its base model, BERT-Large, on five different classification datasets. The largest improvements are on the target domain. Pretrained transformer models, such as CT-BERT, are trained on a specific target domain and can be used for a wide variety of natural language processing tasks, including classification, question-answering and chatbots. CT-BERT is optimised to be used on COVID-19 content, in particular social media posts from Twitter.
6G Communications: A Vision on the Potential Applications
Venue: arXiv cs.NI 2020
Year: 2020

6G communication technology is a revolutionary technology that will revolutionize many technologies and applications. Furthermore, it will be truly AI-driven and will carry on intelligent space. Hence, it will enable Internet of Everything (IoE) which will also impact many technologies and applications. 6G communication technology promises high Quality of Services (QoS) and high Quality of Experiences (QoE). With the combination of IoE and 6G communication technology, number of applications will be exploded in the coming future, particularly, vehicles, drones, homes, cities, hospitals, and so on, and there will be no untouched area. Thence, it is expected that many existing technologies will fully depend on 6G communication technology and enhance their performances. 6G communication technology will prove as game changer communication technology in many fields and will be capable to influence many applications. Therefore, we envision the potential applications of 6G communication technology in the near future.
Machine Learning as a Catalyst for Value-Based Health Care
Venue: arXiv cs.CY 2020
Year: 2020

In this manuscript, we present an argument that machine learning, a subfield of artificial intelligence, can drive improvement in value-based health care through reducing error in clinical decision making. Much of what has been previously published on machine learning in medicine represent single-use or proof-of-concept cases, as well as broad reviews of the advantages and limitations of machine learning. It is timely to look at the broader strategy for artificial intelligence implementation in medicine and emphasize how machine learning can positively influence value-based care.
BAHULAM: Distributed Data Analytics on Secure Enclaves
Venue: arXiv cs.DC 2020
Year: 2020

This is a survey of some of the currently available frameworks (opensource/commercial) in order to run distributed data applications(Hadoop, Spark) on secure enclaves. Intel, AMD, Amazon support secure enclaves on their systems Intel-SGX, AMD Memory Encryption, AWS Nitro Enclaves respectively. Keystone is an open-source framework for architecting Trusted Execution Environments and isolation.
Blockchain Architecture forAuditing Automation and TrustBuilding in Public Markets
Venue: arXiv cs.CR 2020
Year: 2020

Business transactions by public firms are required to be reported, verified, and audited periodically, which is traditionally a labor-intensive and time-consuming process. To streamline this procedure, we design FutureAB (Future Auditing Blockchain) which aims to automate the reporting and auditing process, thereby allowing auditors to focus on discretionary accounts to better detect and prevent fraud. We demonstrate how distributed-ledger technologies build investor trust and disrupt the auditing industry. Our multi-functional design indicates that auditing firms can automate transaction verification without the need for a trusted third party by collaborating and sharing their information while preserving data privacy (commitment scheme) and security (immutability). We also explore how smart contracts and wallets facilitate the computerization and implementation of our system on Ethereum. Finally, performance evaluation reveals the efficacy and scalability of FutureAB in terms of both encryption (0.012 seconds per transaction) and verification (0.001 seconds per transaction).
Watching the Watchers: Nonce-based Inverse Surveillance to Remotely Detect Monitoring
Venue: arXiv cs.NI 2020
Year: 2020

Internet users and service providers do not often know when traffic is being watched but desire a way to determine when, where, and by whom. We present NOISE, the Nonce Observatory for Inverse Surveillance of Eavesdroppers, a method and system that detects monitoring by disseminating nonces - unique, pseudorandom values - in traffic and seeing if they are acted upon unexpectedly, indicating that the nonce-laden traffic is being monitored. Specifically, we embed 64-bit nonces innocuously into IPv6 addresses and disseminate these nonces Internet-wide using a modified traceroute-like tool that makes each outbound probe's source address unique. We continually monitor for subsequent nonce propagation, i.e., activity or interest involving these nonces, e.g., via packet capture on our system's infrastructure. Across three experiments and four months, NOISE detects monitoring more than 200k times, ostensibly in 268 networks, for probes destined for 437 networks. Our results reveal: (a) data collection for security incident handling, (b) traffic information being shared with third parties, and (c) eavesdropping in or near a large commercial peering exchange.
Complex Societies and the Growth of the Law
Venue: arXiv cs.SI 2020
Year: 2020

While a large number of informal factors influence how people interact, modern societies rely upon law as a primary mechanism to formally control human behaviour. How legal rules impact societal development depends on the interplay between two types of actors: the people who create the rules and the people to which the rules potentially apply. We hypothesise that an increasingly diverse and interconnected society might create increasingly diverse and interconnected rules, and assert that legal networks provide a useful lens through which to observe the interaction between law and society. To evaluate these propositions, we present a novel and generalizable model of statutory materials as multidimensional, time-evolving document networks. Applying this model to the federal legislation of the United States and Germany, we find impressive expansion in the size and complexity of laws over the past two and a half decades. We investigate the sources of this development using methods from network science and natural language processing. To allow for cross-country comparisons over time, we algorithmically reorganise the legislative materials of the United States and Germany into cluster families that reflect legal topics. This reorganisation reveals that the main driver behind the growth of the law in both jurisdictions is the expansion of the welfare state, backed by an expansion of the tax state.
Benchmarking neural embeddings for link prediction in knowledge graphs under semantic and structural changes
Venue: arXiv cs.AI 2020
Year: 2020

Recently, link prediction algorithms based on neural embeddings have gained tremendous popularity in the Semantic Web community, and are extensively used for knowledge graph completion. While algorithmic advances have strongly focused on efficient ways of learning embeddings, fewer attention has been drawn to the different ways their performance and robustness can be evaluated. In this work we propose an open-source evaluation pipeline, which benchmarks the accuracy of neural embeddings in situations where knowledge graphs may experience semantic and structural changes. We define relation-centric connectivity measures that allow us to connect the link prediction capacity to the structure of the knowledge graph. Such an evaluation pipeline is especially important to simulate the accuracy of embeddings for knowledge graphs that are expected to be frequently updated.
Trustworthy Edge Computing through Blockchains
Venue: arXiv cs.DC 2020
Year: 2020

Edge computing draws a lot of recent research interests because of the performance improvement by offloading many workloads from the remote data center to nearby edge nodes. Nonetheless, one open challenge of this emerging paradigm lies in the potential security issues on edge nodes and end devices, e.g., sensors and controllers. This paper proposes a cooperative protocol, namely DEAN, across edge nodes to prevent data manipulation, and to allow fair data sharing with quick recovery under resource constraints of limited storage, computing, and network capacity. Specifically, DEAN leverages a parallel mechanism equipped with three independent core components, effectively achieving low resource consumption while allowing secured parallel block processing on edge nodes. We have implemented a system prototype based on DEAN and experimentally verified its effectiveness with a comparison with three popular blockchain implementations: Ethereum, Parity, and Hyperledger Fabric. Experimental results show that the system prototype exhibits high resilience to arbitrary failures: the percentile of trusty nodes is much higher than the required 50\% in most cases. Performance-wise, DEAN-based blockchain implementation outperforms the state-of-the-art blockchain systems with up to $25\times$ higher throughput and $18\times$ lower latency on 1,000 nodes.
Momentum with Variance Reduction for Nonconvex Composition Optimization
Venue: arXiv cs.LG 2020
Year: 2020

Composition optimization is widely-applied in nonconvex machine learning. Various advanced stochastic algorithms that adopt momentum and variance reduction techniques have been developed for composition optimization. However, these algorithms do not fully exploit both techniques to accelerate the convergence and are lack of convergence guarantee in nonconvex optimization. This paper complements the existing literature by developing various momentum schemes with SPIDER-based variance reduction for non-convex composition optimization. In particular, our momentum design requires less number of proximal mapping evaluations per-iteration than that required by the existing Katyusha momentum. Furthermore, our algorithm achieves near-optimal sample complexity results in both non-convex finite-sum and online composition optimization and achieves a linear convergence rate under the gradient dominant condition. Numerical experiments demonstrate that our algorithm converges significantly faster than existing algorithms in nonconvex composition optimization.
On the Information Plane of Autoencoders
Venue: arXiv cs.LG 2020
Year: 2020

The training dynamics of hidden layers in deep learning are poorly understood in theory. Recently, the Information Plane (IP) was proposed to analyze them, which is based on the information-theoretic concept of mutual information (MI). The Information Bottleneck (IB) theory predicts that layers maximize relevant information and compress irrelevant information. Due to the limitations in MI estimation from samples, there is an ongoing debate about the properties of the IP for the supervised learning case. In this work, we derive a theoretical convergence for the IP of autoencoders. The theory predicts that ideal autoencoders with a large bottleneck layer size do not compress input information, whereas a small size causes compression only in the encoder layers. For the experiments, we use a Gram-matrix based MI estimator recently proposed in the literature. We propose a new rule to adjust its parameters that compensates scale and dimensionality effects. Using our proposed rule, we obtain experimental IPs closer to the theory. Our theoretical IP for autoencoders could be used as a benchmark to validate new methods to estimate MI in neural networks. In this way, experimental limitations could be recognized and corrected, helping with the ongoing debate on the supervised learning case.
A flexible, extensible software framework for model compression based on the LC algorithm
Venue: arXiv cs.LG 2020
Year: 2020

We propose a software framework based on the ideas of the Learning-Compression (LC) algorithm, that allows a user to compress a neural network or other machine learning model using different compression schemes with minimal effort. Currently, the supported compressions include pruning, quantization, low-rank methods (including automatically learning the layer ranks), and combinations of those, and the user can choose different compression types for different parts of a neural network. The LC algorithm alternates two types of steps until convergence: a learning (L) step, which trains a model on a dataset (using an algorithm such as SGD); and a compression (C) step, which compresses the model parameters (using a compression scheme such as low-rank or quantization). This decoupling of the "machine learning" aspect from the "signal compression" aspect means that changing the model or the compression type amounts to calling the corresponding subroutine in the L or C step, respectively. The library fully supports this by design, which makes it flexible and extensible. This does not come at the expense of performance: the runtime needed to compress a model is comparable to that of training the model in the first place; and the compressed model is competitive in terms of prediction accuracy and compression ratio with other algorithms (which are often specialized for specific models or compression schemes). The library is written in Python and PyTorch and available in Github.
Technologies and Workflow of Creative Coding Projects: Examples from the Google DevArt Competition
Venue: arXiv cs.HC 2020
Year: 2020

Recently, many artists and creative technologists created computer programming as a goal to create something expressive instead of something functional. In this paper, I analyzed 18 creative coding projects from the Google DevArt competition and summarized the critical technologies, types of content, and key workflows of these creative coding projects. The paper also discusses the potential research opportunities for creative coding and art technologies.
Data Freshness in Leader-Based Replicated Storage
Venue: arXiv cs.DC 2020
Year: 2020

Leader-based data replication improves consistency in highly available distributed storage systems via sequential writes to the leader nodes. After a write has been committed by the leaders, follower nodes are written by a multicast mechanism and are only guaranteed to be eventually consistent. With Age of Information (AoI) as the freshness metric, we characterize how the number of leaders affects the freshness of the data retrieved by an instantaneous read query. In particular, we derive the average age of a read query for a deterministic model for the leader writing time and a probabilistic model for the follower writing time. We obtain a closed-form expression for the average age for exponentially distributed follower writing time. Our numerical results show that, depending on the relative speed of the write operation to the two groups of nodes, there exists an optimal number of leaders which minimizes the average age of the retrieved data, and that this number increases as the relative speed of writing on leaders increases.
Optimal Path Planning for Automated Dimensional Inspection of Free-Form Surfaces
Venue: arXiv cs.RO 2020
Year: 2020

Structural dimensional inspection is vital for the process monitoring, quality control, and fault diagnosis in the mass production of auto bodies. Comparing with the non-contact measurement, the high-precision five-axis measuring machine with the touch-trigger probe is a preferred choice for data collection. It can assist manufacturers in making accurate inspection quickly. As the increase of free-form surfaces and diverse surface orientations in auto body design, existing inspection approaches cannot capture some new critical features in the curvature of auto bodies in an efficient way. Therefore, we need to develop new path planning methods for automated dimensional inspection of free-form surfaces. This paper proposes an optimal path planning system for automated programming of measuring point inspection by incorporating probe rotations and effective collision detection. Specifically, the methodological contributions include: (i) a dynamic searching volume-based algorithm is developed to detect potential collisions in the local path between measurement points; (ii) a local path generation method is proposed with the integration of the probe trajectory and the stylus rotation. Then, the inspection time matrix is proposed to quantify the measuring time of diverse local paths; (iii) an optimization approach of the global inspection path for all critical points on the auto body is developed to minimize the total inspection time. A case study has been conducted on an auto body to verify the performance of the proposed method. Results show that the collision-free path for the free-form auto body could be generated automatically with off-line programming, and the proposed method produces about 40% fewer dummy points and needs 32% less movement time in the auto body inspection process.
Research Challenges for Heterogeneous CPS Design
Venue: arXiv cs.SE 2020
Year: 2020

Heterogeneous computing is widely used at all levels of computing from data center to edge due to its power/performance characteristics. However, heterogeneity presents challenges. Interoperability---the management of workloads across heterogeneous resources---requires more careful design than is the case for homogeneous platforms. Cyber-physical systems present additional challenges. This article considers research challenges in heterogeneous CPS design, including interoperability, physical modeling, models of computation, self-awareness and adaptation, architecture, and scheduling.
Neural Stochastic Block Model & Scalable Community-Based Graph Learning
Venue: arXiv cs.SI 2020
Year: 2020

This paper proposes a novel scalable community-based neural framework for graph learning. The framework learns the graph topology through the task of community detection and link prediction by optimizing with our proposed joint SBM loss function, which results from a non-trivial adaptation of the likelihood function of the classic Stochastic Block Model (SBM). Compared with SBM, our framework is flexible, naturally allows soft labels and digestion of complex node attributes. The main goal is efficient valuation of complex graph data, therefore our design carefully aims at accommodating large data, and ensures there is a single forward pass for efficient evaluation. For large graph, it remains an open problem of how to efficiently leverage its underlying structure for various graph learning tasks. Previously it can be heavy work. With our community-based framework, this becomes less difficult and allows the task models to basically plug-in-and-play and perform joint training. We currently look into two particular applications, the graph alignment and the anomalous correlation detection, and discuss how to make use of our framework to tackle both problems. Extensive experiments are conducted to demonstrate the effectiveness of our approach. We also contributed tweaks of classic techniques which we find helpful for performance and scalability. For example, 1) the GAT+, an improved design of GAT (Graph Attention Network), the scaled-cosine similarity, and a unified implementation of the convolution/attention based and the random-walk based neural graph models.
Education and Technology In Kuwait Democratic Practice
Venue: arXiv cs.CY 2020
Year: 2020

This study discusses the effects of the new technology on the democratic system by reviewing the political history of democracy in philosophic view. The study explores the philosophical discussion of democracy and its components that is designed by most influential philosophers whom most of them agreed on the importance of education and knowledge to be the two main components. The study connects the democratic progress in the State of Kuwait to its roots, namely knowledge and education.
Causal Modeling of Twitter Activity During COVID-19
Venue: arXiv cs.SI 2020
Year: 2020

Understanding the characteristics of public attention and sentiment is an essential prerequisite for appropriate crisis management during adverse health events. This is even more crucial during a pandemic such as COVID-19, as primary responsibility of risk management is not centralized to a single institution, but distributed across society. While numerous studies utilize Twitter data in descriptive or predictive context during COVID-19 pandemic, causal modeling of public attention has not been investigated. In this study, we propose a causal inference approach to discover and quantify causal relationships between pandemic characteristics (e.g. number of infections and deaths) and Twitter activity as well as public sentiment. Our results show that the proposed method can successfully capture the epidemiological domain knowledge and identify variables that affect public attention and sentiment. We believe our work contributes to the field of infodemiology by distinguishing events that correlate with public attention from events that cause public attention.
Logical Inferences with Comparatives and Generalized Quantifiers
Venue: arXiv cs.CL 2020
Year: 2020

Comparative constructions pose a challenge in Natural Language Inference (NLI), which is the task of determining whether a text entails a hypothesis. Comparatives are structurally complex in that they interact with other linguistic phenomena such as quantifiers, numerals, and lexical antonyms. In formal semantics, there is a rich body of work on comparatives and gradable expressions using the notion of degree. However, a logical inference system for comparatives has not been sufficiently developed for use in the NLI task. In this paper, we present a compositional semantics that maps various comparative constructions in English to semantic representations via Combinatory Categorial Grammar (CCG) parsers and combine it with an inference system based on automated theorem proving. We evaluate our system on three NLI datasets that contain complex logical inferences with comparatives, generalized quantifiers, and numerals. We show that the system outperforms previous logic-based systems as well as recent deep learning-based models.