OpenRaiser/Intern-Atlas huggingface dataset:
Local and non-local dependency learning and emergence of rule-like representations in speech data by Deep Convolutional Generative Adversarial Networks
Venue: arXiv cs.CL 2021
Year: 2021

This paper argues that training GANs on local and non-local dependencies in speech data offers insights into how deep neural networks discretize continuous data and how symbolic-like rule-based morphophonological processes emerge in a deep convolutional architecture. Acquisition of speech has recently been modeled as a dependency between latent space and data generated by GANs in Beguš (2020b; arXiv:2006.03965), who models learning of a simple local allophonic distribution. We extend this approach to test learning of local and non-local phonological processes that include approximations of morphological processes. We further parallel outputs of the model to results of a behavioral experiment where human subjects are trained on the data used for training the GAN network. Four main conclusions emerge: (i) the networks provide useful information for computational models of speech acquisition even if trained on a comparatively small dataset of an artificial grammar learning experiment; (ii) local processes are easier to learn than non-local processes, which matches both behavioral data in human subjects and typology in the world's languages. This paper also proposes (iii) how we can actively observe the network's progress in learning and explore the effect of training steps on learning representations by keeping latent space constant across different training steps. Finally, this paper shows that (iv) the network learns to encode the presence of a prefix with a single latent variable; by interpolating this variable, we can actively observe the operation of a non-local phonological process. The proposed technique for retrieving learning representations has general implications for our understanding of how GANs discretize continuous speech data and suggests that rule-like generalizations in the training data are represented as an interaction between variables in the network's latent space.
Learning Optimal Representations with the Decodable Information Bottleneck
Venue: arXiv cs.LG 2021
Year: 2021

We address the question of characterizing and finding optimal representations for supervised learning. Traditionally, this question has been tackled using the Information Bottleneck, which compresses the inputs while retaining information about the targets, in a decoder-agnostic fashion. In machine learning, however, our goal is not compression but rather generalization, which is intimately linked to the predictive family or decoder of interest (e.g. linear classifier). We propose the Decodable Information Bottleneck (DIB) that considers information retention and compression from the perspective of the desired predictive family. As a result, DIB gives rise to representations that are optimal in terms of expected test performance and can be estimated with guarantees. Empirically, we show that the framework can be used to enforce a small generalization gap on downstream classifiers and to predict the generalization ability of neural networks.
Rank/Select Queries over Mutable Bitmaps
Venue: arXiv cs.DS 2021
Year: 2021

The problem of answering rank/select queries over a bitmap is of utmost importance for many succinct data structures. When the bitmap does not change, many solutions exist in the theoretical and practical side. In this work we consider the case where one is allowed to modify the bitmap via a flip(i) operation that toggles its i-th bit. By adapting and properly extending some results concerning prefix-sum data structures, we present a practical solution to the problem, tailored for modern CPU instruction sets. Compared to the state-of-the-art, our solution improves runtime with no space degradation. Moreover, it does not incur in a significant runtime penalty when compared to the fastest immutable indexes, while providing even lower space overhead.
Self-Organizing Software Models for the Internet of Things
Venue: arXiv cs.SE 2021
Year: 2021

The Internet of Things (IoT) envisions the integration of physical objects into software systems for automating crucial aspects of our lives, such as healthcare, security, agriculture, and city management. Although the vision is promising, with the rapid advancement of hardware and communication technologies, IoT systems are becoming increasingly dynamic, large, and complex to the extent that manual management becomes infeasible. Thus, it is of paramount importance to provide software engineering foundations for constructing autonomic IoT systems. In this paper, we introduce a novel paradigm referred to as self-organizing software models in which IoT software systems are not explicitly programmed, but emerge in a decentralized manner during system operation, with minimal or without human intervention. We particularly present an overview of these models by including their definition, motivation, research challenges, and potential directions.
Privacy-Preserving Dynamic Personalized Pricing with Demand Learning
Venue: arXiv cs.CR 2021
Year: 2021

The prevalence of e-commerce has made detailed customers' personal information readily accessible to retailers, and this information has been widely used in pricing decisions. When involving personalized information, how to protect the privacy of such information becomes a critical issue in practice. In this paper, we consider a dynamic pricing problem over $T$ time periods with an \emph{unknown} demand function of posted price and personalized information. At each time $t$, the retailer observes an arriving customer's personal information and offers a price. The customer then makes the purchase decision, which will be utilized by the retailer to learn the underlying demand function. There is potentially a serious privacy concern during this process: a third party agent might infer the personalized information and purchase decisions from price changes from the pricing system. Using the fundamental framework of differential privacy from computer science, we develop a privacy-preserving dynamic pricing policy, which tries to maximize the retailer revenue while avoiding information leakage of individual customer's information and purchasing decisions. To this end, we first introduce a notion of \emph{anticipating} $(\varepsilon, δ)$-differential privacy that is tailored to dynamic pricing problem. Our policy achieves both the privacy guarantee and the performance guarantee in terms of regret. Roughly speaking, for $d$-dimensional personalized information, our algorithm achieves the expected regret at the order of $\tilde{O}(\varepsilon^{-1} \sqrt{d^3 T})$, when the customers' information is adversarially chosen. For stochastic personalized information, the regret bound can be further improved to $\tilde{O}(\sqrt{d^2T} + \varepsilon^{-2} d^2)$
Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges
Venue: arXiv cs.NI 2021
Year: 2021

The Internet of Things (IoT) will be ripe for the deployment of novel machine learning algorithms for both network and application management. However, given the presence of massively distributed and private datasets, it is challenging to use classical centralized learning algorithms in the IoT. To overcome this challenge, federated learning can be a promising solution that enables on-device machine learning without the need to migrate the private end-user data to a central cloud. In federated learning, only learning model updates are transferred between end-devices and the aggregation server. Although federated learning can offer better privacy preservation than centralized machine learning, it has still privacy concerns. In this paper, first, we present the recent advances of federated learning towards enabling federated learning-powered IoT applications. A set of metrics such as sparsification, robustness, quantization, scalability, security, and privacy, is delineated in order to rigorously evaluate the recent advances. Second, we devise a taxonomy for federated learning over IoT networks. Third, we propose two IoT use cases of dispersed federated learning that can offer better privacy preservation than federated learning. Finally, we present several open research challenges with their possible solutions.
Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
Venue: arXiv cs.LG 2021
Year: 2021

In this paper we introduce a provably stable architecture for Neural Ordinary Differential Equations (ODEs) which achieves non-trivial adversarial robustness under white-box adversarial attacks even when the network is trained naturally. For most existing defense methods withstanding strong white-box attacks, to improve robustness of neural networks, they need to be trained adversarially, hence have to strike a trade-off between natural accuracy and adversarial robustness. Inspired by dynamical system theory, we design a stabilized neural ODE network named SONet whose ODE blocks are skew-symmetric and proved to be input-output stable. With natural training, SONet can achieve comparable robustness with the state-of-the-art adversarial defense methods, without sacrificing natural accuracy. Even replacing only the first layer of a ResNet by such a ODE block can exhibit further improvement in robustness, e.g., under PGD-20 ($\ell_\infty=0.031$) attack on CIFAR-10 dataset, it achieves 91.57\% and natural accuracy and 62.35\% robust accuracy, while a counterpart architecture of ResNet trained with TRADES achieves natural and robust accuracy 76.29\% and 45.24\%, respectively. To understand possible reasons behind this surprisingly good result, we further explore the possible mechanism underlying such an adversarial robustness. We show that the adaptive stepsize numerical ODE solver, DOPRI5, has a gradient masking effect that fails the PGD attacks which are sensitive to gradient information of training loss; on the other hand, it cannot fool the CW attack of robust gradients and the SPSA attack that is gradient-free. This provides a new explanation that the adversarial robustness of ODE-based networks mainly comes from the obfuscated gradients in numerical ODE solvers.
Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
Venue: arXiv cs.LG 2021
Year: 2021

Deep reinforcement learning has recently seen huge success across multiple areas in the robotics domain. Owing to the limitations of gathering real-world data, i.e., sample inefficiency and the cost of collecting it, simulation environments are utilized for training the different agents. This not only aids in providing a potentially infinite data source, but also alleviates safety concerns with real robots. Nonetheless, the gap between the simulated and real worlds degrades the performance of the policies once the models are transferred into real robots. Multiple research efforts are therefore now being directed towards closing this sim-to-real gap and accomplish more efficient policy transfer. Recent years have seen the emergence of multiple methods applicable to different domains, but there is a lack, to the best of our knowledge, of a comprehensive review summarizing and putting into context the different methods. In this survey paper, we cover the fundamental background behind sim-to-real transfer in deep reinforcement learning and overview the main methods being utilized at the moment: domain randomization, domain adaptation, imitation learning, meta-learning and knowledge distillation. We categorize some of the most relevant recent works, and outline the main application scenarios. Finally, we discuss the main opportunities and challenges of the different approaches and point to the most promising directions.
Partially Observable Minimum-Age Scheduling: The Greedy Policy
Venue: arXiv cs.IT 2021
Year: 2021

This paper studies the minimum-age scheduling problem in a wireless sensor network where an access point (AP) monitors the state of an object via a set of sensors. The freshness of the sensed state, measured by the age-of-information (AoI), varies at different sensors and is not directly observable to the AP. The AP has to decide which sensor to query/sample in order to get the most updated state information of the object (i.e., the state information with the minimum AoI). In this paper, we formulate the minimum-age scheduling problem as a multi-armed bandit problem with partially observable arms and explore the greedy policy to minimize the expected AoI sampled over an infinite horizon. To analyze the performance of the greedy policy, we 1) put forth a relaxed greedy policy that decouples the sampling processes of the arms, 2) formulate the sampling process of each arm as a partially observable Markov decision process (POMDP), and 3) derive the average sampled AoI under the relaxed greedy policy as a sum of the average AoI sampled from individual arms. Numerical and simulation results validate that the relaxed greedy policy is an excellent approximation to the greedy policy in terms of the expected AoI sampled over an infinite horizon.
From Twitter to Traffic Predictor: Next-Day Morning Traffic Prediction Using Social Media Data
Venue: arXiv cs.SI 2021
Year: 2021

The effectiveness of traditional traffic prediction methods is often extremely limited when forecasting traffic dynamics in early morning. The reason is that traffic can break down drastically during the early morning commute, and the time and duration of this break-down vary substantially from day to day. Early morning traffic forecast is crucial to inform morning-commute traffic management, but they are generally challenging to predict in advance, particularly by midnight. In this paper, we propose to mine Twitter messages as a probing method to understand the impacts of people's work and rest patterns in the evening/midnight of the previous day to the next-day morning traffic. The model is tested on freeway networks in Pittsburgh as experiments. The resulting relationship is surprisingly simple and powerful. We find that, in general, the earlier people rest as indicated from Tweets, the more congested roads will be in the next morning. The occurrence of big events in the evening before, represented by higher or lower tweet sentiment than normal, often implies lower travel demand in the next morning than normal days. Besides, people's tweeting activities in the night before and early morning are statistically associated with congestion in morning peak hours. We make use of such relationships to build a predictive framework which forecasts morning commute congestion using people's tweeting profiles extracted by 5 am or as late as the midnight prior to the morning. The Pittsburgh study supports that our framework can precisely predict morning congestion, particularly for some road segments upstream of roadway bottlenecks with large day-to-day congestion variation. Our approach considerably outperforms those existing methods without Twitter message features, and it can learn meaningful representation of demand from tweeting profiles that offer managerial insights.
Research and Education Towards Smart and Sustainable World
Venue: arXiv cs.AI 2021
Year: 2021

We propose a vision for directing research and education in the ICT field. Our Smart and Sustainable World vision targets at prosperity for the people and the planet through better awareness and control of both human-made and natural environment. The needs of the society, individuals, and industries are fulfilled with intelligent systems that sense their environment, make proactive decisions on actions advancing their goals, and perform the actions on the environment. We emphasize artificial intelligence, feedback loops, human acceptance and control, intelligent use of basic resources, performance parameters, mission-oriented interdisciplinary research, and a holistic systems view complementing the conventional analytical reductive view as a research paradigm especially for complex problems. To serve a broad audience, we explain these concepts and list the essential literature. We suggest planning research and education by specifying, in a step-wise manner, scenarios, performance criteria, system models, research problems and education content, resulting in common goals and a coherent project portfolio as well as education curricula. Research and education produce feedback to support evolutionary development and encourage creativity in research. Finally, we propose concrete actions for realizing this approach.
Macroeconomic forecasting through news, emotions and narrative
Venue: arXiv cs.CY 2021
Year: 2021

This study proposes a new method of incorporating emotions from newspaper articles into macroeconomic forecasts, attempting to forecast industrial production and consumer prices leveraging narrative and sentiment from global newspapers. For the most part, existing research includes positive and negative tone only to improve macroeconomic forecasts, focusing predominantly on large economies such as the US. These works use mainly anglophone sources of narrative, thus not capturing the entire complexity of the multitude of emotions contained in global news articles. This study expands the existing body of research by incorporating a wide array of emotions from newspapers around the world - extracted from the Global Database of Events, Language and Tone (GDELT) - into macroeconomic forecasts. We present a thematic data filtering methodology based on a bi-directional long short term memory neural network (Bi-LSTM) for extracting emotion scores from GDELT and demonstrate its effectiveness by comparing results for filtered and unfiltered data. We model industrial production and consumer prices across a diverse range of economies using an autoregressive framework, and find that including emotions from global newspapers significantly improves forecasts compared to three autoregressive benchmark models. We complement our forecasts with an interpretability analysis on distinct groups of emotions and find that emotions associated with happiness and anger have the strongest predictive power for the variables we predict.
NatCat: Weakly Supervised Text Classification with Naturally Annotated Resources
Venue: arXiv cs.CL 2021
Year: 2021

We describe NatCat, a large-scale resource for text classification constructed from three data sources: Wikipedia, Stack Exchange, and Reddit. NatCat consists of document-category pairs derived from manual curation that occurs naturally within online communities. To demonstrate its usefulness, we build general purpose text classifiers by training on NatCat and evaluate them on a suite of 11 text classification tasks (CatEval), reporting large improvements compared to prior work. We benchmark different modeling choices and resource combinations and show how tasks benefit from particular NatCat data sources.
Rain-Code Fusion : Code-to-code ConvLSTM Forecasting Spatiotemporal Precipitation
Venue: arXiv cs.LG 2021
Year: 2021

Recently, flood damage has become a social problem owing to unexperienced weather conditions arising from climate change. An immediate response to heavy rain is important for the mitigation of economic losses and also for rapid recovery. Spatiotemporal precipitation forecasts may enhance the accuracy of dam inflow prediction, more than 6 hours forward for flood damage mitigation. However, the ordinary ConvLSTM has the limitation of predictable range more than 3-timesteps in real-world precipitation forecasting owing to the irreducible bias between target prediction and ground-truth value. This paper proposes a rain-code approach for spatiotemporal precipitation code-to-code forecasting. We propose a novel rainy feature that represents a temporal rainy process using multi-frame fusion for the timestep reduction. We perform rain-code studies with various term ranges based on the standard ConvLSTM. We applied to a dam region within the Japanese rainy term hourly precipitation data, under 2006 to 2019 approximately 127 thousands hours, every year from May to October. We apply the radar analysis hourly data on the central broader region with an area of 136 x 148 km2 . Finally we have provided sensitivity studies between the rain-code size and hourly accuracy within the several forecasting range.
Between Shapes, Using the Hausdorff Distance
Venue: arXiv cs.CG 2021
Year: 2021

Given two shapes $A$ and $B$ in the plane with Hausdorff distance $1$, is there a shape $S$ with Hausdorff distance $1/2$ to and from $A$ and $B$? The answer is always yes, and depending on convexity of $A$ and/or $B$, $S$ may be convex, connected, or disconnected. We show that our result can be generalised to give an interpolated shape between $A$ and $B$ for any interpolation variable $α$ between $0$ and $1$, and prove that the resulting morph has a bounded rate of change with respect to $α$. Finally, we explore a generalization of the concept of a Hausdorff middle to more than two input sets. We show how to approximate or compute this middle shape, and that the properties relating to the connectedness of the Hausdorff middle extend from the case with two input sets. We also give bounds on the Hausdorff distance between the middle set and the input.
GCNNMatch: Graph Convolutional Neural Networks for Multi-Object Tracking via Sinkhorn Normalization
Venue: arXiv cs.CV 2021
Year: 2021

This paper proposes a novel method for online Multi-Object Tracking (MOT) using Graph Convolutional Neural Network (GCNN) based feature extraction and end-to-end feature matching for object association. The Graph based approach incorporates both appearance and geometry of objects at past frames as well as the current frame into the task of feature learning. This new paradigm enables the network to leverage the "context" information of the geometry of objects and allows us to model the interactions among the features of multiple objects. Another central innovation of our proposed framework is the use of the Sinkhorn algorithm for end-to-end learning of the associations among objects during model training. The network is trained to predict object associations by taking into account constraints specific to the MOT task. Experimental results demonstrate the efficacy of the proposed approach in achieving top performance on the MOT '15, '16, '17 and '20 Challenges among state-of-the-art online approaches. The code is available at https://github.com/IPapakis/GCNNMatch.
Low-Rank and Sparse Enhanced Tucker Decomposition for Tensor Completion
Venue: arXiv cs.LG 2021
Year: 2021

Tensor completion refers to the task of estimating the missing data from an incomplete measurement or observation, which is a core problem frequently arising from the areas of big data analysis, computer vision, and network engineering. Due to the multidimensional nature of high-order tensors, the matrix approaches, e.g., matrix factorization and direct matricization of tensors, are often not ideal for tensor completion and recovery. In this paper, we introduce a unified low-rank and sparse enhanced Tucker decomposition model for tensor completion. Our model possesses a sparse regularization term to promote a sparse core tensor of the Tucker decomposition, which is beneficial for tensor data compression. Moreover, we enforce low-rank regularization terms on factor matrices of the Tucker decomposition for inducing the low-rankness of the tensor with a cheap computational cost. Numerically, we propose a customized ADMM with enough easy subproblems to solve the underlying model. It is remarkable that our model is able to deal with different types of real-world data sets, since it exploits the potential periodicity and inherent correlation properties appeared in tensors. A series of computational experiments on real-world data sets, including internet traffic data sets, color images, and face recognition, demonstrate that our model performs better than many existing state-of-the-art matricization and tensorization approaches in terms of achieving higher recovery accuracy.