OpenRaiser/Intern-Atlas huggingface dataset:
CCS Explorer: Relevance Prediction, Extractive Summarization, and Named Entity Recognition from Clinical Cohort Studies
Venue: arXiv cs.CL 2022
Year: 2022

Clinical Cohort Studies (CCS), such as randomized clinical trials, are a great source of documented clinical research. Ideally, a clinical expert inspects these articles for exploratory analysis ranging from drug discovery for evaluating the efficacy of existing drugs in tackling emerging diseases to the first test of newly developed drugs. However, more than 100 articles are published daily on a single prevalent disease like COVID-19 in PubMed. As a result, it can take days for a physician to find articles and extract relevant information. Can we develop a system to sift through the long list of these articles faster and document the crucial takeaways from each of these articles? In this work, we propose CCS Explorer, an end-to-end system for relevance prediction of sentences, extractive summarization, and patient, outcome, and intervention entity detection from CCS. CCS Explorer is packaged in a web-based graphical user interface where the user can provide any disease name. CCS Explorer then extracts and aggregates all relevant information from articles on PubMed based on the results of an automatically generated query produced on the back-end. For each task, CCS Explorer fine-tunes pre-trained language representation models based on transformers with additional layers. The models are evaluated using two publicly available datasets. CCS Explorer obtains a recall of 80.2%, AUC-ROC of 0.843, and an accuracy of 88.3% on sentence relevance prediction using BioBERT and achieves an average Micro F1-Score of 77.8% on Patient, Intervention, Outcome detection (PIO) using PubMedBERT. Thus, CCS Explorer can reliably extract relevant information to summarize articles, saving time by $\sim \text{660}\times$.
Transfer Learning with Kernel Methods
Venue: arXiv cs.LG 2022
Year: 2022

Transfer learning refers to the process of adapting a model trained on a source task to a target task. While kernel methods are conceptually and computationally simple machine learning models that are competitive on a variety of tasks, it has been unclear how to perform transfer learning for kernel methods. In this work, we propose a transfer learning framework for kernel methods by projecting and translating the source model to the target task. We demonstrate the effectiveness of our framework in applications to image classification and virtual drug screening. In particular, we show that transferring modern kernels trained on large-scale image datasets can result in substantial performance increase as compared to using the same kernel trained directly on the target task. In addition, we show that transfer-learned kernels allow a more accurate prediction of the effect of drugs on cancer cell lines. For both applications, we identify simple scaling laws that characterize the performance of transfer-learned kernels as a function of the number of target examples. We explain this phenomenon in a simplified linear setting, where we are able to derive the exact scaling laws. By providing a simple and effective transfer learning framework for kernel methods, our work enables kernel methods trained on large datasets to be easily adapted to a variety of downstream target tasks.
SADT: Combining Sharpness-Aware Minimization with Self-Distillation for Improved Model Generalization
Venue: arXiv cs.LG 2022
Year: 2022

Methods for improving deep neural network training times and model generalizability consist of various data augmentation, regularization, and optimization approaches, which tend to be sensitive to hyperparameter settings and make reproducibility more challenging. This work jointly considers two recent training strategies that address model generalizability: sharpness-aware minimization, and self-distillation, and proposes the novel training strategy of Sharpness-Aware Distilled Teachers (SADT). The experimental section of this work shows that SADT consistently outperforms previously published training strategies in model convergence time, test-time performance, and model generalizability over various neural architectures, datasets, and hyperparameter settings.
Entity Matching by Pool-based Active Learning
Venue: arXiv cs.LG 2022
Year: 2022

The goal of entity matching is to find the corresponding records representing the same real-world entity from different data sources. At present, in the mainstream methods, rule-based entity matching methods need tremendous domain knowledge. The machine-learning based or deep-learning based entity matching methods need a large number of labeled samples to build the model, which is difficult to achieve in some applications. In addition, learning-based methods are easy to over-fitting, so the quality requirements of training samples are very high. In this paper, we present an active learning method ALMatcher for the entity matching tasks. This method needs to manually label only a small number of valuable samples, and use these samples to build a model with high quality. This paper proposes a hybrid uncertainty as query strategy to find those valuable samples for labeling, which can minimize the number of labeled training samples meanwhile meet the task requirements. The proposed method has been validated on seven data sets in different fields. The experiment shows that ALMatcher uses only a small number of labeled samples and achieves better results compared to existing approaches.
Meta-Learning for Unsupervised Outlier Detection with Optimal Transport
Venue: arXiv cs.LG 2022
Year: 2022

Automated machine learning has been widely researched and adopted in the field of supervised classification and regression, but progress in unsupervised settings has been limited. We propose a novel approach to automate outlier detection based on meta-learning from previous datasets with outliers. Our premise is that the selection of the optimal outlier detection technique depends on the inherent properties of the data distribution. We leverage optimal transport in particular, to find the dataset with the most similar underlying distribution, and then apply the outlier detection techniques that proved to work best for that data distribution. We evaluate the robustness of our approach and find that it outperforms the state of the art methods in unsupervised outlier detection. This approach can also be easily generalized to automate other unsupervised settings.
On Kubernetes-aided Federated Database Systems
Venue: arXiv cs.DC 2022
Year: 2022

Cloud computing has made federated database systems (FDBS) significantly more practical to implement than in the past. As part of a recent Web-based Geographic Information System (WebGIS) project, we are employing cloud-native technologies (from the container ecosystem) to develop a federated database (DB) infrastructure, to help manage and utilise the distributed and various geospatial data. Unfortunately, there seem to be inherent challenges and complexity of applying the container and Kubernetes technologies to building and running DB systems. Considering that most of the geospatial and theme data are pre-obtained and fixed in our WebGIS project, we decided to focus on the read-only user queries and still resort to Kubernetes to implement an FDBS instance to use. Unlike the de facto practices (e.g., using the StatefulSets mechanism, extending Kuberentes APIs, or employing KubeFed), our solution for Kubernetes-aided FDBS simplifies the tech stack by investigating the fractal object of federated data management, inclusively containerising DB instances, and using the lightweight Deployment mechanism to handle stateless DB containers. Overall, this research not only reveals an easy-to-implement approach to constructing read-only components in a fully-fledged FDBS, but also proposes and demonstrates a novel methodology for FDBS investigations.
The future is different: Large pre-trained language models fail in prediction tasks
Venue: arXiv cs.CL 2022
Year: 2022

Large pre-trained language models (LPLM) have shown spectacular success when fine-tuned on downstream supervised tasks. Yet, it is known that their performance can drastically drop when there is a distribution shift between the data used during training and that used at inference time. In this paper we focus on data distributions that naturally change over time and introduce four new REDDIT datasets, namely the WALLSTREETBETS, ASKSCIENCE, THE DONALD, and POLITICS sub-reddits. First, we empirically demonstrate that LPLM can display average performance drops of about 88% (in the best case!) when predicting the popularity of future posts from sub-reddits whose topic distribution changes with time. We then introduce a simple methodology that leverages neural variational dynamic topic models and attention mechanisms to infer temporal language model representations for regression tasks. Our models display performance drops of only about 40% in the worst cases (2% in the best ones) when predicting the popularity of future posts, while using only about 7% of the total number of parameters of LPLM and providing interpretable representations that offer insight into real-world events, like the GameStop short squeeze of 2021
Reasoning on Property Graphs with Graph Generating Dependencies
Venue: arXiv cs.DB 2022
Year: 2022

Graph Generating Dependencies (GGDs) informally express constraints between two (possibly different) graph patterns which enforce relationships on both graph's data (via property value constraints) and its structure (via topological constraints). Graph Generating Dependencies (GGDs) can express tuple- and equality-generating dependencies on property graphs, both of which find broad application in graph data management. In this paper, we discuss the reasoning behind GGDs. We propose algorithms to solve the satisfiability, implication, and validation problems for GGDs and analyze their complexity. To demonstrate the practical use of GGDs, we propose an algorithm which finds inconsistencies in data through validation of GGDs. Our experiments show that even though the validation of GGDs has high computational complexity, GGDs can be used to find data inconsistencies in a feasible execution time on both synthetic and real-world data.
Using coevolution and substitution of the fittest for health and well-being recommender systems
Venue: arXiv cs.NE 2022
Year: 2022

This research explores substitution of the fittest (SF), a technique designed to counteract the problem of disengagement in two-population competitive coevolutionary genetic algorithms. SF is domain-independent and requires no calibration. We first perform a controlled comparative evaluation of SF's ability to maintain engagement and discover optimal solutions in a minimal toy domain. Experimental results demonstrate that SF is able to maintain engagement better than other techniques in the literature. We then address the more complex real-world problem of evolving recommendations for health and well-being. We introduce a coevolutionary extension of EvoRecSys, a previously published evolutionary recommender system. We demonstrate that SF is able to maintain engagement better than other techniques in the literature, and the resultant recommendations using SF are higher quality and more diverse than those produced by EvoRecSys.
Two classes of subfield codes of linear codes
Venue: arXiv cs.IT 2022
Year: 2022

Recently, subfiled codes of linear code over GF$ (q) $ with good parameters were studied, and many optimal subfield codes were obtained. In this paper, Our mainly motivation is to generlize the results of the subfield codes of hyperoval in Ding and Heng (Finite Fields Their Appl. 56, 308-331 (2019)), and generlize the results of two families of subfield codes in Xiang and Yin (Cryptogr. Commun. 13(1), 117-127 (2021)) to $ p $-ary where $ p $ is odd. We get the parameters and weight distribution of these subfield codes. At the same time, the parameters of their dual codes are also studied. When $ m=1 $, The dual codes of these subfield codes are almost MDS code, when $ m>1 $ and $ p $ odd, these dual codes are dimension-optimal with respect to the sphere-backing bound.
Should I disclose my dataset? Caveats between reproducibility and individual data rights
Venue: arXiv cs.CY 2022
Year: 2022

Natural language processing techniques have helped domain experts solve legal problems. Digital availability of court documents increases possibilities for researchers, who can access them as a source for building datasets -- whose disclosure is aligned with good reproducibility practices in computational research. Large and digitized court systems, such as the Brazilian one, are prone to be explored in that sense. However, personal data protection laws impose restrictions on data exposure and state principles about which researchers should be mindful. Special caution must be taken in cases with human rights violations, such as gender discrimination, over which we elaborate as an example of interest. We present legal and ethical considerations on the issue, as well as guidelines for researchers dealing with this kind of data and deciding whether to disclose it.
MDC Enhanced IoT Networks: Network Modeling and Performance Analysis
Venue: arXiv cs.NI 2022
Year: 2022

As a promising architecture, Mobile Data Collector (MDC) enhanced Internet of Things (IoT) exhibits broad prospects in efficient data collection and data aggregation especially for sparse deployment scenarios. Combining the tools from queueing theory and stochastic geometry, we propose an analytical framework to study the network performance of an MDC enhanced IoT network, in terms of coverage probability, end-to-end delay and energy consumption. We derive the closed-form expressions for average contact and inter-contact time between a sensor and its associated MDC. By modeling the data collection system between a sensor and its associated MDCs as an M/G/1 queue system with vacations and general limited (G-limited) service, we first derive the queueing delay at the tagged sensor, and further obtain the end-to-end delay. The proposed analytical framework enables us to quantify the effect on network performance of key system parameters, such as MDC velocity, packet arrival rate, densities of sensors and MDCs, and contact radius. This study reveals that the MDC velocity has little impact on the coverage probability, and provides guidelines to minimize the end-to-end delay by optimizing the density and contact radius of sensors, and the velocity and density of MDCs.
Leveraging Graph-based Cross-modal Information Fusion for Neural Sign Language Translation
Venue: arXiv cs.CL 2022
Year: 2022

Sign Language (SL), as the mother tongue of the deaf community, is a special visual language that most hearing people cannot understand. In recent years, neural Sign Language Translation (SLT), as a possible way for bridging communication gap between the deaf and the hearing people, has attracted widespread academic attention. We found that the current mainstream end-to-end neural SLT models, which tries to learning language knowledge in a weakly supervised manner, could not mine enough semantic information under the condition of low data resources. Therefore, we propose to introduce additional word-level semantic knowledge of sign language linguistics to assist in improving current end-to-end neural SLT models. Concretely, we propose a novel neural SLT model with multi-modal feature fusion based on the dynamic graph, in which the cross-modal information, i.e. text and video, is first assembled as a dynamic graph according to their correlation, and then the graph is processed by a multi-modal graph encoder to generate the multi-modal embeddings for further usage in the subsequent neural translation models. To the best of our knowledge, we are the first to introduce graph neural networks, for fusing multi-modal information, into neural sign language translation models. Moreover, we conducted experiments on a publicly available popular SLT dataset RWTH-PHOENIX-Weather-2014T. and the quantitative experiments show that our method can improve the model.
No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration
Venue: arXiv cs.CV 2022
Year: 2022

Recognizing who is speaking in a crowded scene is a key challenge towards the understanding of the social interactions going on within. Detecting speaking status from body movement alone opens the door for the analysis of social scenes in which personal audio is not obtainable. Video and wearable sensors make it possible recognize speaking in an unobtrusive, privacy-preserving way. When considering the video modality, in action recognition problems, a bounding box is traditionally used to localize and segment out the target subject, to then recognize the action taking place within it. However, cross-contamination, occlusion, and the articulated nature of the human body, make this approach challenging in a crowded scene. Here, we leverage articulated body poses for subject localization and in the subsequent speech detection stage. We show that the selection of local features around pose keypoints has a positive effect on generalization performance while also significantly reducing the number of local features considered, making for a more efficient method. Using two in-the-wild datasets with different viewpoints of subjects, we investigate the role of cross-contamination in this effect. We additionally make use of acceleration measured through wearable sensors for the same task, and present a multimodal approach combining both methods.
GLINKX: A Scalable Unified Framework For Homophilous and Heterophilous Graphs
Venue: arXiv cs.LG 2022
Year: 2022

In graph learning, there have been two predominant inductive biases regarding graph-inspired architectures: On the one hand, higher-order interactions and message passing work well on homophilous graphs and are leveraged by GCNs and GATs. Such architectures, however, cannot easily scale to large real-world graphs. On the other hand, shallow (or node-level) models using ego features and adjacency embeddings work well in heterophilous graphs. In this work, we propose a novel scalable shallow method -- GLINKX -- that can work both on homophilous and heterophilous graphs. GLINKX leverages (i) novel monophilous label propagations, (ii) ego/node features, (iii) knowledge graph embeddings as positional embeddings, (iv) node-level training, and (v) low-dimensional message passing. Formally, we prove novel error bounds and justify the components of GLINKX. Experimentally, we show its effectiveness on several homophilous and heterophilous datasets.
Revisiting Heterophily in Graph Convolution Networks by Learning Representations Across Topological and Feature Spaces
Venue: arXiv cs.LG 2022
Year: 2022

Graph convolution networks (GCNs) have been enormously successful in learning representations over several graph-based machine learning tasks. Specific to learning rich node representations, most of the methods have solely relied on the homophily assumption and have shown limited performance on the heterophilous graphs. While several methods have been developed with new architectures to address heterophily, we argue that by learning graph representations across two spaces i.e., topology and feature space GCNs can address heterophily. In this work, we experimentally demonstrate the performance of the proposed GCN framework over semi-supervised node classification task on both homophilous and heterophilous graph benchmarks by learning and combining representations across the topological and the feature spaces.
Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructions
Venue: arXiv cs.AI 2022
Year: 2022

The adoption of pre-trained language models to generate action plans for embodied agents is a promising research strategy. However, execution of instructions in real or simulated environments requires verification of the feasibility of actions as well as their relevance to the completion of a goal. We propose a new method that combines a language model and reinforcement learning for the task of building objects in a Minecraft-like environment according to the natural language instructions. Our method first generates a set of consistently achievable sub-goals from the instructions and then completes associated sub-tasks with a pre-trained RL policy. The proposed method formed the RL baseline at the IGLU 2022 competition.
Semantic Pivoting Model for Effective Event Detection
Venue: arXiv cs.CL 2022
Year: 2022

Event Detection, which aims to identify and classify mentions of event instances from unstructured articles, is an important task in Natural Language Processing (NLP). Existing techniques for event detection only use homogeneous one-hot vectors to represent the event type classes, ignoring the fact that the semantic meaning of the types is important to the task. Such an approach is inefficient and prone to overfitting. In this paper, we propose a Semantic Pivoting Model for Effective Event Detection (SPEED), which explicitly incorporates prior information during training and captures semantically meaningful correlations between input and events. Experimental results show that our proposed model achieves state-of-the-art performance and outperforms the baselines in multiple settings without using any external resources.
VIINTER: View Interpolation with Implicit Neural Representations of Images
Venue: arXiv cs.CV 2022
Year: 2022

We present VIINTER, a method for view interpolation by interpolating the implicit neural representation (INR) of the captured images. We leverage the learned code vector associated with each image and interpolate between these codes to achieve viewpoint transitions. We propose several techniques that significantly enhance the interpolation quality. VIINTER signifies a new way to achieve view interpolation without constructing 3D structure, estimating camera poses, or computing pixel correspondence. We validate the effectiveness of VIINTER on several multi-view scenes with different types of camera layout and scene composition. As the development of INR of images (as opposed to surface or volume) has centered around tasks like image fitting and super-resolution, with VIINTER, we show its capability for view interpolation and offer a promising outlook on using INR for image manipulation tasks.
On the Interaction Between Differential Privacy and Gradient Compression in Deep Learning
Venue: arXiv cs.LG 2022
Year: 2022

While differential privacy and gradient compression are separately well-researched topics in machine learning, the study of interaction between these two topics is still relatively new. We perform a detailed empirical study on how the Gaussian mechanism for differential privacy and gradient compression jointly impact test accuracy in deep learning. The existing literature in gradient compression mostly evaluates compression in the absence of differential privacy guarantees, and demonstrate that sufficiently high compression rates reduce accuracy. Similarly, existing literature in differential privacy evaluates privacy mechanisms in the absence of compression, and demonstrates that sufficiently strong privacy guarantees reduce accuracy. In this work, we observe while gradient compression generally has a negative impact on test accuracy in non-private training, it can sometimes improve test accuracy in differentially private training. Specifically, we observe that when employing aggressive sparsification or rank reduction to the gradients, test accuracy is less affected by the Gaussian noise added for differential privacy. These observations are explained through an analysis how differential privacy and compression effects the bias and variance in estimating the average gradient. We follow this study with a recommendation on how to improve test accuracy under the context of differentially private deep learning and gradient compression. We evaluate this proposal and find that it can reduce the negative impact of noise added by differential privacy mechanisms on test accuracy by up to 24.6%, and reduce the negative impact of gradient sparsification on test accuracy by up to 15.1%.