OpenRaiser/Intern-Atlas huggingface dataset:
Out-of-Manifold Regularization in Contextual Embedding Space for Text Classification
Venue: arXiv cs.CL 2021
Year: 2021

Recent studies on neural networks with pre-trained weights (i.e., BERT) have mainly focused on a low-dimensional subspace, where the embedding vectors computed from input words (or their contexts) are located. In this work, we propose a new approach to finding and regularizing the remainder of the space, referred to as out-of-manifold, which cannot be accessed through the words. Specifically, we synthesize the out-of-manifold embeddings based on two embeddings obtained from actually-observed words, to utilize them for fine-tuning the network. A discriminator is trained to detect whether an input embedding is located inside the manifold or not, and simultaneously, a generator is optimized to produce new embeddings that can be easily identified as out-of-manifold by the discriminator. These two modules successfully collaborate in a unified and end-to-end manner for regularizing the out-of-manifold. Our extensive evaluation on various text classification benchmarks demonstrates the effectiveness of our approach, as well as its good compatibility with existing data augmentation techniques which aim to enhance the manifold.
SAT-Based Rigorous Explanations for Decision Lists
Venue: arXiv cs.AI 2021
Year: 2021

Decision lists (DLs) find a wide range of uses for classification problems in Machine Learning (ML), being implemented in a number of ML frameworks. DLs are often perceived as interpretable. However, building on recent results for decision trees (DTs), we argue that interpretability is an elusive goal for some DLs. As a result, for some uses of DLs, it will be important to compute (rigorous) explanations. Unfortunately, and in clear contrast with the case of DTs, this paper shows that computing explanations for DLs is computationally hard. Motivated by this result, the paper proposes propositional encodings for computing abductive explanations (AXps) and contrastive explanations (CXps) of DLs. Furthermore, the paper investigates the practical efficiency of a MARCO-like approach for enumerating explanations. The experimental results demonstrate that, for DLs used in practical settings, the use of SAT oracles offers a very efficient solution, and that complete enumeration of explanations is most often feasible.
A cost-benefit analysis of cross-lingual transfer methods
Venue: arXiv cs.CL 2021
Year: 2021

An effective method for cross-lingual transfer is to fine-tune a bilingual or multilingual model on a supervised dataset in one language and evaluating it on another language in a zero-shot manner. Translating examples at training time or inference time are also viable alternatives. However, there are costs associated with these methods that are rarely addressed in the literature. In this work, we analyze cross-lingual methods in terms of their effectiveness (e.g., accuracy), development and deployment costs, as well as their latencies at inference time. Our experiments on three tasks indicate that the best cross-lingual method is highly task-dependent. Finally, by combining zero-shot and translation methods, we achieve the state-of-the-art in two of the three datasets used in this work. Based on these results, we question the need for manually labeled training data in a target language. Code and translated datasets are available at https://github.com/unicamp-dl/cross-lingual-analysis
Multi-Objective Optimisation of Cortical Spiking Neural Networks With Genetic Algorithms
Venue: arXiv cs.NE 2021
Year: 2021

Spiking neural networks (SNNs) communicate through the all-or-none spiking activity of neurons. However, fitting the large number of SNN model parameters to observed neural activity patterns, for example, in biological experiments, remains a challenge. Previous work using genetic algorithm (GA) optimisation on a specific efficient SNN model, using the Izhikevich neuronal model, was limited to a single parameter and objective. This work applied a version of GA, called non-dominated sorting GA (NSGA-III), to demonstrate the feasibility of performing multi-objective optimisation on the same SNN, focusing on searching network connectivity parameters to achieve target firing rates of excitatory and inhibitory neuronal types, including across different network connectivity sparsity. We showed that NSGA-III could readily optimise for various firing rates. Notably, when the excitatory neural firing rates were higher than or equal to that of inhibitory neurons, the errors were small. Moreover, when connectivity sparsity was considered as a parameter to be optimised, the optimal solutions required sparse network connectivity. We also found that for excitatory neural firing rates lower than that of inhibitory neurons, the errors were generally larger. Overall, we have successfully demonstrated the feasibility of implementing multi-objective GA optimisation on network parameters of recurrent and sparse SNN.
The academic wanderer: structure of collaboration network and relation with research performance
Venue: arXiv cs.SI 2021
Year: 2021

Thanks to the widespread availability of large-scale datasets on scholarly outputs, science itself has come under the microscope with the aim of capturing a quantitative understanding of its workings. In this study, we leverage well-established cognitive models coming from anthropology in order to characterise the personal network of collaborations between scientists, i.e., the network considered from the standpoint of each individual researcher (referred to as ego network), in terms of the cognitive investment they devote to the different collaborations. Building upon these models, we study the interplay between the structure of academic collaborations, academic performance, and academic mobility at different career stages. We take into account both purely academic mobility (i.e., the number of affiliation changes) and geographical mobility (i.e., physical relocations to different countries). For our investigation, we rely on a dataset comprising the geo-referenced publications of a group of 81,500 authors extracted from Scopus, one of the biggest repositories of academic knowledge. Our main finding is that there is a clear correlation between the structure of co-authorship ego networks and academic performance indices: the more one publishes and the higher their impact, the larger their collaboration network. However, we observe a capacity bound effect, whereby, beyond a certain point, higher performances become increasingly less correlated with large collaboration networks. We also find that international academic migrants are better at growing their networks than researchers that only migrate within the same country, but the latter seem to be better in exploiting their collaboration to achieve higher impact. High academic mobility does not appear to translate into better academic performance or larger collaboration networks.
VICE: Visual Identification and Correction of Neural Circuit Errors
Venue: arXiv cs.CV 2021
Year: 2021

A connectivity graph of neurons at the resolution of single synapses provides scientists with a tool for understanding the nervous system in health and disease. Recent advances in automatic image segmentation and synapse prediction in electron microscopy (EM) datasets of the brain have made reconstructions of neurons possible at the nanometer scale. However, automatic segmentation sometimes struggles to segment large neurons correctly, requiring human effort to proofread its output. General proofreading involves inspecting large volumes to correct segmentation errors at the pixel level, a visually intensive and time-consuming process. This paper presents the design and implementation of an analytics framework that streamlines proofreading, focusing on connectivity-related errors. We accomplish this with automated likely-error detection and synapse clustering that drives the proofreading effort with highly interactive 3D visualizations. In particular, our strategy centers on proofreading the local circuit of a single cell to ensure a basic level of completeness. We demonstrate our framework's utility with a user study and report quantitative and subjective feedback from our users. Overall, users find the framework more efficient for proofreading, understanding evolving graphs, and sharing error correction strategies.
Delivering Gigabit Capacities to Passenger Trains Tales from an Operator on the Road to 5G
Venue: arXiv cs.NI 2021
Year: 2021

Delivering reliable and high-capacity Internet connectivity to high-speed train users is a challenge. Modern railway cars act as Faraday cages and a typical train consist comprises several hundreds of users moving at high velocity. Furthermore, with the global availability of fourth generation (4G) Long Term Evolution (LTE), user expectations have dramatically increased: it is expected to be online anytime and anywhere. Demand for mobile high-capacity is being driven by video and music streaming services, for lower latency and higher availability by gaming, and for more reliability and even uplink capacity by mission critical applications. Finally, the life-cycle of the railway industry is much longer than for telecommunications, which makes supporting 5G challenging. In this paper, we survey the challenges associated with delivering high-capacity connectivity to train users, describe potential options, and highlight how a leading western European operator is tackling these challenges and preparing for 5G and beyond.
Face Attributes as Cues for Deep Face Recognition Understanding
Venue: arXiv cs.CV 2021
Year: 2021

Deeply learned representations are the state-of-the-art descriptors for face recognition methods. These representations encode latent features that are difficult to explain, compromising the confidence and interpretability of their predictions. Most attempts to explain deep features are visualization techniques that are often open to interpretation. Instead of relying only on visualizations, we use the outputs of hidden layers to predict face attributes. The obtained performance is an indicator of how well the attribute is implicitly learned in that layer of the network. Using a variable selection technique, we also analyze how these semantic concepts are distributed inside each layer, establishing the precise location of relevant neurons for each attribute. According to our experiments, gender, eyeglasses and hat usage can be predicted with over 96% accuracy even when only a single neural output is used to predict each attribute. These performances are less than 3 percentage points lower than the ones achieved by deep supervised face attribute networks. In summary, our experiments show that, inside DCNNs optimized for face identification, there exists latent neurons encoding face attributes almost as accurately as DCNNs optimized for these attributes.
Measuring the User Satisfaction in a Recommendation Interface with Multiple Carousels
Venue: arXiv cs.IR 2021
Year: 2021

It is common for video-on-demand and music streaming services to adopt a user interface composed of several recommendation lists, i.e. widgets or swipeable carousels, each generated according to a specific criterion or algorithm (e.g. most recent, top popular, recommended for you, editors' choice, etc.). Selecting the appropriate combination of carousel has significant impact on user satisfaction. A crucial aspect of this user interface is that to measure the relevance a new carousel for the user it is not sufficient to account solely for its individual quality. Instead, it should be considered that other carousels will already be present in the interface. This is not considered by traditional evaluation protocols for recommenders systems, in which each carousel is evaluated in isolation, regardless of (i) which other carousels are displayed to the user and (ii) the relative position of the carousel with respect to other carousels. Hence, we propose a two-dimensional evaluation protocol for a carousel setting that will measure the quality of a recommendation carousel based on how much it improves upon the quality of an already available set of carousels. Our evaluation protocol takes into account also the position bias, i.e. users do not explore the carousels sequentially, but rather concentrate on the top-left corner of the screen. We report experiments on the movie domain and notice that under a carousel setting the definition of which criteria has to be preferred to generate a list of recommended items changes with respect to what is commonly understood.
Analyzing Images for Music Recommendation
Venue: arXiv cs.MM 2021
Year: 2021

Experiencing images with suitable music can greatly enrich the overall user experience. The proposed image analysis method treats an artwork image differently from a photograph image. Automatic image classification is performed using deep-learning based models. An illustrative analysis showcasing the ability of our deep-models to inherently learn and utilize perceptually relevant features when classifying artworks is also presented. The Mean Opinion Score (MOS) obtained from subjective assessments of the respective image and recommended music pairs supports the effectiveness of our approach.
Joint learning of multiple Granger causal networks via non-convex regularizations: Inference of group-level brain connectivity
Venue: arXiv cs.LG 2021
Year: 2021

This paper considers joint learning of multiple sparse Granger graphical models to discover underlying common and differential Granger causality (GC) structures across multiple time series. This can be applied to drawing group-level brain connectivity inferences from a homogeneous group of subjects or discovering network differences among groups of signals collected under heterogeneous conditions. By recognizing that the GC of a single multivariate time series can be characterized by common zeros of vector autoregressive (VAR) lag coefficients, a group sparse prior is included in joint regularized least-squares estimations of multiple VAR models. Group-norm regularizations based on group- and fused-lasso penalties encourage a decomposition of multiple networks into a common GC structure, with other remaining parts defined in individual-specific networks. Prior information about sparseness and sparsity patterns of desired GC networks are incorporated as relative weights, while a non-convex group norm in the penalty is proposed to enhance the accuracy of network estimation in low-sample settings. Extensive numerical results on simulations illustrated our method's improvements over existing sparse estimation approaches on GC network sparsity recovery. Our methods were also applied to available resting-state fMRI time series from the ADHD-200 data sets to learn the differences of causality mechanisms, called effective brain connectivity, between adolescents with ADHD and typically developing children. Our analysis revealed that parts of the causality differences between the two groups often resided in the orbitofrontal region and areas associated with the limbic system, which agreed with clinical findings and data-driven results in previous studies.
Are Convolutional Neural Networks or Transformers more like human vision?
Venue: arXiv cs.CV 2021
Year: 2021

Modern machine learning models for computer vision exceed humans in accuracy on specific visual recognition tasks, notably on datasets like ImageNet. However, high accuracy can be achieved in many ways. The particular decision function found by a machine learning system is determined not only by the data to which the system is exposed, but also the inductive biases of the model, which are typically harder to characterize. In this work, we follow a recent trend of in-depth behavioral analyses of neural network models that go beyond accuracy as an evaluation metric by looking at patterns of errors. Our focus is on comparing a suite of standard Convolutional Neural Networks (CNNs) and a recently-proposed attention-based network, the Vision Transformer (ViT), which relaxes the translation-invariance constraint of CNNs and therefore represents a model with a weaker set of inductive biases. Attention-based networks have previously been shown to achieve higher accuracy than CNNs on vision tasks, and we demonstrate, using new metrics for examining error consistency with more granularity, that their errors are also more consistent with those of humans. These results have implications both for building more human-like vision models, as well as for understanding visual object recognition in humans.
On the Distributional Properties of Adaptive Gradients
Venue: arXiv cs.LG 2021
Year: 2021

Adaptive gradient methods have achieved remarkable success in training deep neural networks on a wide variety of tasks. However, not much is known about the mathematical and statistical properties of this family of methods. This work aims at providing a series of theoretical analyses of its statistical properties justified by experiments. In particular, we show that when the underlying gradient obeys a normal distribution, the variance of the magnitude of the \textit{update} is an increasing and bounded function of time and does not diverge. This work suggests that the divergence of variance is not the cause of the need for warm up of the Adam optimizer, contrary to what is believed in the current literature.
Using Ethnographic Methods to Classify the Human Experience in Medicine: A Case Study of the Presence Ontology
Venue: arXiv cs.IT 2021
Year: 2021

Objective Although social and environmental factors are central to provider patient interactions, the data that reflect these factors can be incomplete, vague, and subjective. We sought to create a conceptual framework to describe and classify data about presence, the domain of interpersonal connection in medicine. Methods Our top down approach for ontology development based on the concept of relationality included 1) broad survey of social sciences literature and systematic literature review of more than 20,000 articles around interpersonal connection in medicine, 3) relational ethnography of clinical encounters (5 pilot, 27 full) and 4) interviews about relational work with 40 medical and nonmedical professionals. We formalized the model using the Web Ontology Language in the Protege ontology editor. We iteratively evaluated and refined the Presence Ontology through manual expert review and automated annotation of literature. Results and Discussion The Presence Ontology facilitates the naming and classification of concepts that would otherwise be vague. Our model categorizes contributors to healthcare encounters and factors such as Communication, Emotions, Tools, and Environment. Ontology evaluation indicated that Cognitive Models (both patients explanatory models and providers caregiving approaches) influenced encounters and were subsequently incorporated. We show how ethnographic methods based in relationality can aid the representation of experiential concepts (e.g., empathy, trust). Our ontology could support informatics applications to improve healthcare such annotation of videotaped encounters, clinical instruments to measure presence, or EHR based reminders for providers. Conclusion The Presence Ontology provides a model for using ethnographic approaches to classify interpersonal data.
Annotation Uncertainty in the Context of Grammatical Change
Venue: arXiv cs.CL 2021
Year: 2021

This paper elaborates on the notion of uncertainty in the context of annotation in large text corpora, specifically focusing on (but not limited to) historical languages. Such uncertainty might be due to inherent properties of the language, for example, linguistic ambiguity and overlapping categories of linguistic description, but could also be caused by lacking annotation expertise. By examining annotation uncertainty in more detail, we identify the sources and deepen our understanding of the nature and different types of uncertainty encountered in daily annotation practice. Moreover, some practical implications of our theoretical findings are also discussed. Last but not least, this article can be seen as an attempt to reconcile the perspectives of the main scientific disciplines involved in corpus projects, linguistics and computer science, to develop a unified view and to highlight the potential synergies between these disciplines.
An Ergonomic Interaction Workspace Analysis Method for the Optimal Design of a Surgical Master Manipulator
Venue: arXiv cs.RO 2021
Year: 2021

Master control console is a place where robots collaborate with humans in a shared environment. To this end, ergonomics is an important aspect to be considered. With ergonomic design, the surgeons can feel more comfortable to conduct the surgical tasks with higher efficiency, and the quality of the teleoperated robotic surgery can be improved. In this paper, an Ergonomic Interaction Workspace Analysis method is proposed to optimize master manipulators and fulfil ergonomics consideration for designing a master manipulator for teleoperated robotic surgery.
Storage Aware Routing for Generalized Delay Tolerant Networks
Venue: arXiv cs.NI 2021
Year: 2021

This paper presents a novel storage aware routing (STAR) protocol designed to provide a general networking solution over a broad range of wired and wireless usage scenarios. STAR enables routing policies which adapt seamlessly from a well-connected wired network to a disconnected wireless network. STAR uses a 2-Dimensional routing metric composed of a short and a long term route cost and storage availability on downstream routers to make store or forward routing decisions. Temporary in-network storage is preferred over forwarding along a path that is slower than average and opportunistic transmission is encouraged when faster than average routes become available. Results from ns2 based simulations show that STAR achieves $40-50\%$ higher throughput compared to OLSR in mobile vehicular and DTN scenarios and does $12-20\%$ better than OLSR in the static mesh case. Experimental evaluation of STAR on the ORBIT testbed validates the protocol implementation, and demonstrates significant performance improvements with 25\% higher peak throughput compared to OLSR in a wireless mesh network.
Towards a Predictive Processing Implementation of the Common Model of Cognition
Venue: arXiv cs.AI 2021
Year: 2021

In this article, we present a cognitive architecture that is built from powerful yet simple neural models. Specifically, we describe an implementation of the common model of cognition grounded in neural generative coding and holographic associative memory. The proposed system creates the groundwork for developing agents that learn continually from diverse tasks as well as model human performance at larger scales than what is possible with existant cognitive architectures.