OpenRaiser/Intern-Atlas huggingface dataset:
A narrowing of AI research?
Venue: arXiv cs.CY 2022
Year: 2022

The arrival of deep learning techniques able to infer patterns from large datasets has dramatically improved the performance of Artificial Intelligence (AI) systems. Deep learning's rapid development and adoption, in great part led by large technology companies, has however created concerns about a premature narrowing in the technological trajectory of AI research despite its weaknesses, which include lack of robustness, high environmental costs, and potentially unfair outcomes. We seek to improve the evidence base with a semantic analysis of AI research in arXiv, a popular pre-prints database. We study the evolution of the thematic diversity of AI research, compare the thematic diversity of AI research in academia and the private sector and measure the influence of private companies in AI research through the citations they receive and their collaborations with other institutions. Our results suggest that diversity in AI research has stagnated in recent years, and that AI research involving the private sector tends to be less diverse and more influential than research in academia. We also find that private sector AI researchers tend to specialise in data-hungry and computationally intensive deep learning methods at the expense of research involving other AI methods, research that considers the societal and ethical implications of AI, and applications in sectors like health. Our results provide a rationale for policy action to prevent a premature narrowing of AI research that could constrain its societal benefits, but we note the informational, incentive and scale hurdles standing in the way of such interventions.
SPARC-LDPC Coding for MIMO Massive Unsourced Random Access
Venue: arXiv cs.IT 2022
Year: 2022

A joint sparse-regression-code (SPARC) and low-density-parity-check (LDPC) coding scheme for multiple-input multiple-output (MIMO) massive unsourced random access (URA) is proposed in this paper. Different from the state-of-the-art covariance-based maximum likelihood (CB-ML) detection scheme, we first split users' messages into two parts. The former part is encoded by SPARCs and tasked to recover part of the messages, the corresponding channel coefficients as well as the interleaving patterns by compressed sensing. The latter part is coded by LDPC codes and then interleaved by the interleave-division multiple access (IDMA) scheme. The decoding of the latter part is based on belief propagation (BP) joint with successive interference cancellation (SIC). Numerical results show our scheme outperforms the CB-ML scheme when the number of antennas at the base station is smaller than that of active users. The complexity of our scheme is with the order $\mathcal{O}\left(2^{B_p}ML+\widehat{K}ML\right)$ and lower than the CB-ML scheme. Moreover, our scheme has higher spectral efficiency (nearly $15$ times larger) than CB-ML as we only split messages into two parts.
Implicit Gradient Regularization
Venue: arXiv cs.LG 2022
Year: 2022

Gradient descent can be surprisingly good at optimizing deep neural networks without overfitting and without explicit regularization. We find that the discrete steps of gradient descent implicitly regularize models by penalizing gradient descent trajectories that have large loss gradients. We call this Implicit Gradient Regularization (IGR) and we use backward error analysis to calculate the size of this regularization. We confirm empirically that implicit gradient regularization biases gradient descent toward flat minima, where test errors are small and solutions are robust to noisy parameter perturbations. Furthermore, we demonstrate that the implicit gradient regularization term can be used as an explicit regularizer, allowing us to control this gradient regularization directly. More broadly, our work indicates that backward error analysis is a useful theoretical approach to the perennial question of how learning rate, model size, and parameter regularization interact to determine the properties of overparameterized models optimized with gradient descent.
Sensing Technologies for Crowd Management, Adaptation, and Information Dissemination in Public Transportation Systems: A Review
Venue: arXiv cs.CY 2022
Year: 2022

Management of crowd information in public transportation (PT) systems is crucial, both to foster sustainable mobility, by increasing the user's comfort and satisfaction during normal operation, as well as to cope with emergency situations, such as pandemic crises, as recently experienced with COVID-19 limitations. This paper presents a taxonomy and review of sensing technologies based on Internet of Things (IoT) for real-time crowd analysis, which can be adopted in the different segments of the PT system (buses/trams/trains, railway/metro stations, and bus/tram stops). To discuss such technologies in a clear systematic perspective, we introduce a reference architecture for crowd management, which employs modern information and communication technologies (ICT) in order to: (i) monitor and predict crowding events; (ii) implement crowd-aware policies for real-time and adaptive operation control in intelligent transportation systems (ITSs); (iii) inform in real-time the users of the crowding status of the PT system, by means of electronic displays installed inside vehicles or at bus/tram stops/stations, and/or by mobile transport applications. It is envisioned that the innovative crowd management functionalities enabled by ICT/IoT sensing technologies can be incrementally implemented as an add-on to state-of-the-art ITS platforms, which are already in use by major PT companies operating in urban areas. Moreover, it is argued that, in this new framework, additional services can be delivered to the passengers, such as, e.g., on-line ticketing, vehicle access control and reservation in severely crowded situations, and evolved crowd-aware route planning.
A Robust graph attention network with dynamic adjusted Graph
Venue: arXiv cs.LG 2022
Year: 2022

Graph Attention Networks(GATs) are useful deep learning models to deal with the graph data. However, recent works show that the classical GAT is vulnerable to adversarial attacks. It degrades dramatically with slight perturbations. Therefore, how to enhance the robustness of GAT is a critical problem. Robust GAT(RoGAT) is proposed in this paper to improve the robustness of GAT based on the revision of the attention mechanism. Different from the original GAT, which uses the attention mechanism for different edges but is still sensitive to the perturbation, RoGAT adds an extra dynamic attention score progressively and improves the robustness. Firstly, RoGAT revises the edges weight based on the smoothness assumption which is quite common for ordinary graphs. Secondly, RoGAT further revises the features to suppress features' noise. Then, an extra attention score is generated by the dynamic edge's weight and can be used to reduce the impact of adversarial attacks. Different experiments against targeted and untargeted attacks on citation data on citation data demonstrate that RoGAT outperforms most of the recent defensive methods.
The Shapley Value of Inconsistency Measures for Functional Dependencies
Venue: arXiv cs.DB 2022
Year: 2022

Quantifying the inconsistency of a database is motivated by various goals including reliability estimation for new datasets and progress indication in data cleaning. Another goal is to attribute to individual tuples a level of responsibility to the overall inconsistency, and thereby prioritize tuples in the explanation or inspection of dirt. Therefore, inconsistency quantification and attribution have been a subject of much research in Knowledge Representation and, more recently, in Databases. As in many other fields, a conventional responsibility sharing mechanism is the Shapley value from cooperative game theory. In this paper, we carry out a systematic investigation of the complexity of the Shapley value in common inconsistency measures for functional-dependency (FD) violations. For several measures we establish a full classification of the FD sets into tractable and intractable classes with respect to Shapley-value computation. We also study the complexity of approximation in intractable cases.
MCW-Net: Single Image Deraining with Multi-level Connections and Wide Regional Non-local Blocks
Venue: arXiv cs.CV 2022
Year: 2022

A recent line of convolutional neural network-based works has succeeded in capturing rain streaks. However, difficulties in detailed recovery still remain. In this paper, we present a multi-level connection and wide regional non-local block network (MCW-Net) to properly restore the original background textures in rainy images. Unlike existing encoder-decoder-based image deraining models that improve performance with additional branches, MCW-Net improves performance by maximizing information utilization without additional branches through the following two proposed methods. The first method is a multi-level connection that repeatedly connects multi-level features of the encoder network to the decoder network. Multi-level connection encourages the decoding process to use the feature information of all levels. In multi-level connection, channel-wise attention is considered to learn which level of features is important in the decoding process of the current level. The second method is a wide regional non-local block. As rain streaks primarily exhibit a vertical distribution, we divide the grid of the image into horizontally-wide patches and apply a non-local operation to each region to explore the rich rain-free background information. Experimental results on both synthetic and real-world rainy datasets demonstrate that the proposed model significantly outperforms existing state-of-the-art models. Furthermore, the results of the joint deraining and segmentation experiment prove that our model contributes effectively to other vision tasks.
Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
Venue: arXiv cs.CL 2022
Year: 2022

Pretrained multilingual contextual representations have shown great success, but due to the limits of their pretraining data, their benefits do not apply equally to all language varieties. This presents a challenge for language varieties unfamiliar to these models, whose labeled \emph{and unlabeled} data is too limited to train a monolingual model effectively. We propose the use of additional language-specific pretraining and vocabulary augmentation to adapt multilingual models to low-resource settings. Using dependency parsing of four diverse low-resource language varieties as a case study, we show that these methods significantly improve performance over baselines, especially in the lowest-resource cases, and demonstrate the importance of the relationship between such models' pretraining data and target language varieties.
MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention
Venue: arXiv cs.LG 2022
Year: 2022

Recent advances in neural forecasting have produced major improvements in accuracy for probabilistic demand prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop a novel decoder-encoder attention for context-alignment, improving forecasting accuracy by allowing the network to study its own history based on the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network to learn context-dependent seasonality functions as well as arbitrary holiday distances. Finally we show that the current state of the art MQ-Forecaster (Wen et al., 2017) models display excess variability by failing to leverage previous errors in the forecast to improve accuracy. We propose a novel decoder-self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.
Modeling emotion for human-like behavior in future intelligent robots
Venue: arXiv cs.AI 2022
Year: 2022

Over the past decades, research in cognitive and affective neuroscience has emphasized that emotion is crucial for human intelligence and in fact inseparable from cognition. Concurrently, there has been growing interest in simulating and modeling emotion-related processes in robots and artificial agents. In this opinion paper, our goal is to provide a snapshot of the present landscape in emotion modeling and to show how neuroscience can help advance the current state of the art. We start with an overview of the existing literature on emotion modeling in three areas of research: affective computing, social robotics, and neurorobotics. Briefly summarizing the current state of knowledge on natural emotion, we then highlight how existing proposals in artificial emotion do not make sufficient contact with neuroscientific evidence. We conclude by providing a set of principles to help guide future research in artificial emotion and intelligent machines more generally. Overall, we argue that a stronger integration of emotion-related processes in robot models is critical for the design of human-like behavior in future intelligent machines. Such integration not only will contribute to the development of autonomous social machines capable of tackling real-world problems but would contribute to advancing understanding of human emotion.
Binary Neural Networks for Memory-Efficient and Effective Visual Place Recognition in Changing Environments
Venue: arXiv cs.CV 2022
Year: 2022

Visual place recognition (VPR) is a robot's ability to determine whether a place was visited before using visual data. While conventional hand-crafted methods for VPR fail under extreme environmental appearance changes, those based on convolutional neural networks (CNNs) achieve state-of-the-art performance but result in heavy runtime processes and model sizes that demand a large amount of memory. Hence, CNN-based approaches are unsuitable for resource-constrained platforms, such as small robots and drones. In this paper, we take a multi-step approach of decreasing the precision of model parameters, combining it with network depth reduction and fewer neurons in the classifier stage to propose a new class of highly compact models that drastically reduces the memory requirements and computational effort while maintaining state-of-the-art VPR performance. To the best of our knowledge, this is the first attempt to propose binary neural networks for solving the visual place recognition problem effectively under changing conditions and with significantly reduced resource requirements. Our best-performing binary neural network, dubbed FloppyNet, achieves comparable VPR performance when considered against its full-precision and deeper counterparts while consuming 99% less memory and increasing the inference speed seven times.
A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
Venue: arXiv cs.LG 2022
Year: 2022

We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a $γ^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discounting ($γ^t$) for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective $(γ= 1)$ where $γ^t$ disappears naturally $(1^t = 1)$. We then propose to interpret the discounting in critic in terms of a bias-variance-representation trade-off and provide supporting empirical results. In the second scenario, we consider optimizing a discounted objective ($γ< 1$) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.
EGMM: an Evidential Version of the Gaussian Mixture Model for Clustering
Venue: arXiv cs.LG 2022
Year: 2022

The Gaussian mixture model (GMM) provides a simple yet principled framework for clustering, with properties suitable for statistical inference. In this paper, we propose a new model-based clustering algorithm, called EGMM (evidential GMM), in the theoretical framework of belief functions to better characterize cluster-membership uncertainty. With a mass function representing the cluster membership of each object, the evidential Gaussian mixture distribution composed of the components over the powerset of the desired clusters is proposed to model the entire dataset. The parameters in EGMM are estimated by a specially designed Expectation-Maximization (EM) algorithm. A validity index allowing automatic determination of the proper number of clusters is also provided. The proposed EGMM is as simple as the classical GMM, but can generate a more informative evidential partition for the considered dataset. The synthetic and real dataset experiments show that the proposed EGMM performs better than other representative clustering algorithms. Besides, its superiority is also demonstrated by an application to multi-modal brain image segmentation.
Disentangling causal effects for hierarchical reinforcement learning
Venue: arXiv cs.AI 2022
Year: 2022

Exploration and credit assignment under sparse rewards are still challenging problems. We argue that these challenges arise in part due to the intrinsic rigidity of operating at the level of actions. Actions can precisely define how to perform an activity but are ill-suited to describe what activity to perform. Instead, causal effects are inherently composable and temporally abstract, making them ideal for descriptive tasks. By leveraging a hierarchy of causal effects, this study aims to expedite the learning of task-specific behavior and aid exploration. Borrowing counterfactual and normality measures from causal literature, we disentangle controllable effects from effects caused by other dynamics of the environment. We propose CEHRL, a hierarchical method that models the distribution of controllable effects using a Variational Autoencoder. This distribution is used by a high-level policy to 1) explore the environment via random effect exploration so that novel effects are continuously discovered and learned, and to 2) learn task-specific behavior by prioritizing the effects that maximize a given reward function. In comparison to exploring with random actions, experimental results show that random effect exploration is a more efficient mechanism and that by assigning credit to few effects rather than many actions, CEHRL learns tasks more rapidly.
Pointwise Binary Classification with Pairwise Confidence Comparisons
Venue: arXiv cs.LG 2022
Year: 2022

To alleviate the data requirement for training effective binary classifiers in binary classification, many weakly supervised learning settings have been proposed. Among them, some consider using pairwise but not pointwise labels, when pointwise labels are not accessible due to privacy, confidentiality, or security reasons. However, as a pairwise label denotes whether or not two data points share a pointwise label, it cannot be easily collected if either point is equally likely to be positive or negative. Thus, in this paper, we propose a novel setting called pairwise comparison (Pcomp) classification, where we have only pairs of unlabeled data that we know one is more likely to be positive than the other. Firstly, we give a Pcomp data generation process, derive an unbiased risk estimator (URE) with theoretical guarantee, and further improve URE using correction functions. Secondly, we link Pcomp classification to noisy-label learning to develop a progressive URE and improve it by imposing consistency regularization. Finally, we demonstrate by experiments the effectiveness of our methods, which suggests Pcomp is a valuable and practically useful type of pairwise supervision besides the pairwise label.
BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning
Venue: arXiv cs.LG 2022
Year: 2022

Despite their theoretical appealingness, Bayesian neural networks (BNNs) are left behind in real-world adoption, mainly due to persistent concerns on their scalability, accessibility, and reliability. In this work, we develop the BayesAdapter framework to relieve these concerns. In particular, we propose to adapt pre-trained deterministic NNs to be variational BNNs via cost-effective Bayesian fine-tuning. Technically, we develop a modularized implementation for the learning of variational BNNs, and refurbish the generally applicable exemplar reparameterization trick through exemplar parallelization to efficiently reduce the gradient variance in stochastic variational inference. Based on the lightweight Bayesian learning paradigm, we conduct extensive experiments on a variety of benchmarks, and show that our method can consistently induce posteriors with higher quality than competitive baselines, yet significantly reducing training overheads. Code is available at https://github.com/thudzj/ScalableBDL.
Denoising Diffusion Implicit Models
Venue: arXiv cs.LG 2022
Year: 2022

Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a Markovian diffusion process. We construct a class of non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. We empirically demonstrate that DDIMs can produce high quality samples $10 \times$ to $50 \times$ faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, and can perform semantically meaningful image interpolation directly in the latent space.
SHERLock: Self-Supervised Hierarchical Event Representation Learning
Venue: arXiv cs.LG 2022
Year: 2022

Temporal event representations are an essential aspect of learning among humans. They allow for succinct encoding of the experiences we have through a variety of sensory inputs. Also, they are believed to be arranged hierarchically, allowing for an efficient representation of complex long-horizon experiences. Additionally, these representations are acquired in a self-supervised manner. Analogously, here we propose a model that learns temporal representations from long-horizon visual demonstration data and associated textual descriptions, without explicit temporal supervision. Our method produces a hierarchy of representations that align more closely with ground-truth human-annotated events (+15.3) than state-of-the-art unsupervised baselines. Our results are comparable to heavily-supervised baselines in complex visual domains such as Chess Openings, YouCook2 and TutorialVQA datasets. Finally, we perform ablation studies illustrating the robustness of our approach. We release our code and demo visualizations in the Supplementary Material.