OpenRaiser/Intern-Atlas huggingface dataset:
Spreading processes with population heterogeneity over multi-layer networks
Venue: arXiv cs.SI 2022
Year: 2022

It's been controversial whether re-opening school will facilitate viral spread among household communities with mitigation strategies such as mask-wearing in place. In this work, we propose an epidemiological model that explores the viral transmission over the multi-layer contact network composed of the school layer and community layer with population heterogeneity on mask-wearing behavior. We derive analytical expressions for three key epidemiological quantities: the probability of emergence, the epidemic threshold, and the expected epidemic size. In particular, we show how the aforementioned quantities depend on the structure of the multi-layer contact network, viral transmission dynamics, and the distribution of the different types of masks within the population. Through extensive simulations, our analytical results show near-perfect agreement with the simulation results with a limited number of nodes. Utilizing the model, we study the impact of the opening/closure of the school layer on the viral transmission dynamics with various mask-wearing scenarios. Interestingly, we found that it's safe to open the school layer with the proper proportion of good-quality masks in the population. Moreover, we validate the theory of the trade-off between source-control and self-protection over a single layer by Tian et al on our multi-layer setting. We conclude that even on a multi-layer network, it's of great significance to treat the spreading process as two distinct phases in mind when considering mitigation strategies. Besides, we would like to remark that our model of spreading process over multi-layer networks with population heterogeneity can also be applied to various other domains, such as misinformation control.
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
Venue: arXiv cs.CV 2022
Year: 2022

Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the latent space at each guidance step. Instead, we propose to bring the NeRF to the latent space, resulting in a Latent-NeRF. Analyzing our Latent-NeRF, we show that while Text-to-3D models can generate impressive results, they are inherently unconstrained and may lack the ability to guide or enforce a specific 3D structure. To assist and direct the 3D generation, we propose to guide our Latent-NeRF using a Sketch-Shape: an abstract geometry that defines the coarse structure of the desired object. Then, we present means to integrate such a constraint directly into a Latent-NeRF. This unique combination of text and shape guidance allows for increased control over the generation process. We also show that latent score distillation can be successfully applied directly on 3D meshes. This allows for generating high-quality textures on a given geometry. Our experiments validate the power of our different forms of guidance and the efficiency of using latent rendering. Implementation is available at https://github.com/eladrich/latent-nerf
PiPa: Pixel- and Patch-wise Self-supervised Learning for Domain Adaptative Semantic Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

Unsupervised Domain Adaptation (UDA) aims to enhance the generalization of the learned model to other domains. The domain-invariant knowledge is transferred from the model trained on labeled source domain, e.g., video game, to unlabeled target domains, e.g., real-world scenarios, saving annotation expenses. Existing UDA methods for semantic segmentation usually focus on minimizing the inter-domain discrepancy of various levels, e.g., pixels, features, and predictions, for extracting domain-invariant knowledge. However, the primary intra-domain knowledge, such as context correlation inside an image, remains underexplored. In an attempt to fill this gap, we propose a unified pixel- and patch-wise self-supervised learning framework, called PiPa, for domain adaptive semantic segmentation that facilitates intra-image pixel-wise correlations and patch-wise semantic consistency against different contexts. The proposed framework exploits the inherent structures of intra-domain images, which: (1) explicitly encourages learning the discriminative pixel-wise features with intra-class compactness and inter-class separability, and (2) motivates the robust feature learning of the identical patch against different contexts or fluctuations. Extensive experiments verify the effectiveness of the proposed method, which obtains competitive accuracy on the two widely-used UDA benchmarks, i.e., 75.6 mIoU on GTA to Cityscapes and 68.2 mIoU on Synthia to Cityscapes. Moreover, our method is compatible with other UDA approaches to further improve the performance without introducing extra parameters.
HigeNet: A Highly Efficient Modeling for Long Sequence Time Series Prediction in AIOps
Venue: arXiv cs.LG 2022
Year: 2022

Modern IT system operation demands the integration of system software and hardware metrics. As a result, it generates a massive amount of data, which can be potentially used to make data-driven operational decisions. In the basic form, the decision model needs to monitor a large set of machine data, such as CPU utilization, allocated memory, disk and network latency, and predicts the system metrics to prevent performance degradation. Nevertheless, building an effective prediction model in this scenario is rather challenging as the model has to accurately capture the long-range coupling dependency in the Multivariate Time-Series (MTS). Moreover, this model needs to have low computational complexity and can scale efficiently to the dimension of data available. In this paper, we propose a highly efficient model named HigeNet to predict the long-time sequence time series. We have deployed the HigeNet on production in the D-matrix platform. We also provide offline evaluations on several publicly available datasets as well as one online dataset to demonstrate the model's efficacy. The extensive experiments show that training time, resource usage and accuracy of the model are found to be significantly better than five state-of-the-art competing models.
Alzheimer's Diagnosis and Generation-Based Chatbot Using Hierarchical Attention and Transformer
Venue: arXiv cs.CL 2022
Year: 2022

In this paper, we propose a natural language processing architecture that can handle tasks that previously required two models as one model. With a single model, we analyze the language patterns and conversational context of Alzheimer's patients and derive answers from two results: patient classification and chatbot. If the patient's language characteristics are identified by chatbots in daily life, doctors can plan more precise diagnosis and treatment for early diagnosis. The proposed model is used to develop chatbots that replace questionnaires that required experts. There are two natural language processing tasks performed by the model. The first is a 'natural language classification' that indicates with probability whether the patient has an illness, and the second is to generate the next 'answer' of the chatbot to the patient's answer. In the first half, a context vector, which is a characteristic of patient utterance, is extracted through a self-attention neural network. This context vector and chatbot (expert, moderator) questions are entered together into the encoder to obtain a matrix containing the characteristics of the interaction between the questioner and the patient. The vectorized matrix becomes a probability value for classification of patients. Enter the matrix into the decoder with the next answer from the chatbot (the moderator) to generate the next utterance. As a result of learning this structure with DmentiaBank's cookie theft description corpus, it was confirmed that the value of the loss function of the encoder and decoder was significantly reduced and converged. This shows that capturing the speech language pattern of Alzheimer's disease patients can contribute to early diagnosis and longitudinal studies of the disease in the future.
End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English
Venue: arXiv cs.CL 2022
Year: 2022

Automation of on-call customer support relies heavily on accurate and efficient speech-to-intent (S2I) systems. Building such systems using multi-component pipelines can pose various challenges because they require large annotated datasets, have higher latency, and have complex deployment. These pipelines are also prone to compounding errors. To overcome these challenges, we discuss an end-to-end (E2E) S2I model for customer support voicebot task in a bilingual setting. We show how we can solve E2E intent classification by leveraging a pre-trained automatic speech recognition (ASR) model with slight modification and fine-tuning on small annotated datasets. Experimental results show that our best E2E model outperforms a conventional pipeline by a relative ~27% on the F1 score.
Multilevel Transformer For Multimodal Emotion Recognition
Venue: arXiv cs.CL 2022
Year: 2022

Multimodal emotion recognition has attracted much attention recently. Fusing multiple modalities effectively with limited labeled data is a challenging task. Considering the success of pre-trained model and fine-grained nature of emotion expression, it is reasonable to take these two aspects into consideration. Unlike previous methods that mainly focus on one aspect, we introduce a novel multi-granularity framework, which combines fine-grained representation with pre-trained utterance-level representation. Inspired by Transformer TTS, we propose a multilevel transformer model to perform fine-grained multimodal emotion recognition. Specifically, we explore different methods to incorporate phoneme-level embedding with word-level embedding. To perform multi-granularity learning, we simply combine multilevel transformer model with Albert. Extensive experimental results show that both our multilevel transformer model and multi-granularity model outperform previous state-of-the-art approaches on IEMOCAP dataset with text transcripts and speech signal.
How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling
Venue: arXiv cs.CL 2022
Year: 2022

Large pre-trained language models (LMs) have been widely adopted in biomedical and clinical domains, introducing many powerful LMs such as bio-lm and BioELECTRA. However, the applicability of these methods to real clinical use cases is hindered, due to the limitation of pre-trained LMs in processing long textual data with thousands of words, which is a common length for a clinical note. In this work, we explore long-range adaptation from such LMs with Longformer, allowing the LMs to capture longer clinical notes context. We conduct experiments on three n2c2 challenges datasets and a longitudinal clinical dataset from Hong Kong Hospital Authority electronic health record (EHR) system to show the effectiveness and generalizability of this concept, achieving 10\% F1-score improvement. Based on our experiments, we conclude that capturing a longer clinical note interval is beneficial to the model performance, but there are different cut-off intervals to achieve the optimal performance for different target variables. Our code is available at https://github.com/HLTCHKUST/long-biomedical-model.
Fast DistilBERT on CPUs
Venue: arXiv cs.CL 2022
Year: 2022

Transformer-based language models have become the standard approach to solving natural language processing tasks. However, industry adoption usually requires the maximum throughput to comply with certain latency constraints that prevents Transformer models from being used in production. To address this gap, model compression techniques such as quantization and pruning may be used to improve inference efficiency. However, these compression techniques require specialized software to apply and deploy at scale. In this work, we propose a new pipeline for creating and running Fast Transformer models on CPUs, utilizing hardware-aware pruning, knowledge distillation, quantization, and our own Transformer inference runtime engine with optimized kernels for sparse and quantized operators. We demonstrate the efficiency of our pipeline by creating a Fast DistilBERT model showing minimal accuracy loss on the question-answering SQuADv1.1 benchmark, and throughput results under typical production constraints and environments. Our results outperform existing state-of-the-art Neural Magic's DeepSparse runtime performance by up to 50% and up to 4.1x performance speedup over ONNX Runtime. Source code is publicly available at https://github.com/intel/intel-extension-for-transformers.
Detection of fraudulent financial papers by picking a collection of characteristics using optimization algorithms and classification techniques based on squirrels
Venue: arXiv cs.NE 2022
Year: 2022

To produce important investment decisions, investors require financial records and economic information. However, most companies manipulate investors and financial institutions by inflating their financial statements. Fraudulent Financial Activities exist in any monetary or financial transaction scenario, whether physical or electronic. A challenging problem that arises in this domain is the issue that affects and troubles individuals and institutions. This problem has attracted more attention in the field in part owing to the prevalence of financial fraud and the paucity of previous research. For this purpose, in this study, the main approach to solve this problem, an anomaly detection-based approach based on a combination of feature selection based on squirrel optimization pattern and classification methods have been used. The aim is to develop this method to provide a model for detecting anomalies in financial statements using a combination of selected features with the nearest neighbor classifications, neural networks, support vector machine, and Bayesian. Anomaly samples are then analyzed and compared to recommended techniques using assessment criteria. Squirrel optimization's meta-exploratory capability, along with the approach's ability to identify abnormalities in financial data, has been shown to be effective in implementing the suggested strategy. They discovered fake financial statements because of their expertise.
Meta-Learning of Neural State-Space Models Using Data From Similar Systems
Venue: arXiv cs.LG 2022
Year: 2022

Deep neural state-space models (SSMs) provide a powerful tool for modeling dynamical systems solely using operational data. Typically, neural SSMs are trained using data collected from the actual system under consideration, despite the likely existence of operational data from similar systems which have previously been deployed in the field. In this paper, we propose the use of model-agnostic meta-learning (MAML) for constructing deep encoder network-based SSMs, by leveraging a combination of archived data from similar systems (used to meta-train offline) and limited data from the actual system (used for rapid online adaptation). We demonstrate using a numerical example that meta-learning can result in more accurate neural SSM models than supervised- or transfer-learning, despite few adaptation steps and limited online data. Additionally, we show that by carefully partitioning and adapting the encoder layers while fixing the state-transition operator, we can achieve comparable performance to MAML while reducing online adaptation complexity.
Extending the Neural Additive Model for Survival Analysis with EHR Data
Venue: arXiv cs.LG 2022
Year: 2022

With increasing interest in applying machine learning to develop healthcare solutions, there is a desire to create interpretable deep learning models for survival analysis. In this paper, we extend the Neural Additive Model (NAM) by incorporating pairwise feature interaction networks and equip these models with loss functions that fit both proportional and non-proportional extensions of the Cox model. We show that within this extended framework, we can construct non-proportional hazard models, which we call TimeNAM, that significantly improve performance over the standard NAM model architecture on benchmark survival datasets. We apply these model architectures to data from the Electronic Health Record (EHR) database of Seoul National University Hospital Gangnam Center (SNUHGC) to build an interpretable neural network survival model for gastric cancer prediction. We demonstrate that on both benchmark survival analysis datasets, as well as on our gastric cancer dataset, our model architectures yield performance that matches, or surpasses, the current state-of-the-art black-box methods.
Cable Network Management Infrastructure Evolution
Venue: arXiv cs.NI 2022
Year: 2022

An approach to enable advanced troubleshooting, granular analysis and service quality of experience assessment is presented. The use of topology information in the identification of each cable network element along with granular information of the element configuration and health is proposed. This technique covers multiple layers including the service layer. At the physical layers street addresses, taps and amplifiers are used to identify impairment location. All layers are leveraged to measure network and service reliability, service degradation and to quantify quality of experience. A cable infrastructure implementation is described as an example.
General Intelligence Requires Rethinking Exploration
Venue: arXiv cs.AI 2022
Year: 2022

We are at the cusp of a transition from "learning from data" to "learning what data to learn from" as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train our models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration, is a universal aspect of learning in open-ended domains, such as the real world. Although the study of exploration in AI is largely limited to the field of reinforcement learning, we argue that exploration is essential to all learning systems, including supervised learning. We propose the problem of generalized exploration to conceptually unify exploration-driven learning between supervised learning and reinforcement learning, allowing us to highlight key similarities across learning settings and open research challenges. Importantly, generalized exploration serves as a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence.
Evaluating How Fine-tuning on Bimodal Data Effects Code Generation
Venue: arXiv cs.LG 2022
Year: 2022

Despite the increase in popularity of language models for code generation, it is still unknown how training on bimodal coding forums affects a model's code generation performance and reliability. We, therefore, collect a dataset of over 2.2M StackOverflow questions with answers for finetuning. These fine-tuned models have average $pass@k$ improvements of 54.64% and 85.35% on the HumanEval (Chen et al., 2021) and Mostly Basic Program Problems (Austin et al., 2021) tasks, respectively. This regime further decreases the number of generated programs with both syntax and runtime errors. However, we find that at higher temperatures, there are significant decreases to the model's ability to generate runnable programs despite higher $pass@k$ scores, underscoring the need for better methods of incorporating such data that mitigate these side effects. The code can be found https://github.com/gabeorlanski/bimodalcode-generation
Category-Adaptive Label Discovery and Noise Rejection for Multi-label Image Recognition with Partial Positive Labels
Venue: arXiv cs.CV 2022
Year: 2022

As a promising solution of reducing annotation cost, training multi-label models with partial positive labels (MLR-PPL), in which merely few positive labels are known while other are missing, attracts increasing attention. Due to the absence of any negative labels, previous works regard unknown labels as negative and adopt traditional MLR algorithms. To reject noisy labels, recent works regard large loss samples as noise but ignore the semantic correlation different multi-label images. In this work, we propose to explore semantic correlation among different images to facilitate the MLR-PPL task. Specifically, we design a unified framework, Category-Adaptive Label Discovery and Noise Rejection, that discovers unknown labels and rejects noisy labels for each category in an adaptive manner. The framework consists of two complementary modules: (1) Category-Adaptive Label Discovery module first measures the semantic similarity between positive samples and then complement unknown labels with high similarities; (2) Category-Adaptive Noise Rejection module first computes the sample weights based on semantic similarities from different samples and then discards noisy labels with low weights. Besides, we propose a novel category-adaptive threshold updating that adaptively adjusts the threshold, to avoid the time-consuming manual tuning process. Extensive experiments demonstrate that our proposed method consistently outperforms current leading algorithms.
A Graph-Based Customizable Handover Framework for LEO Satellite Networks
Venue: arXiv cs.NI 2022
Year: 2022

Future satellite networks are expected to have thousands of low Earth orbit (LEO) satellites orbiting Earth at very high speeds. User equipment (UE) communicating directly with LEO satellites will experience frequent handovers. Managing the handover process is complicated due to the high frequency of handovers and the availability of multiple LEO satellites as handover targets. In addition, as the status of the communication link between a UE and an LEO satellite varies in accordance with the visibility period of the satellite, initiating handovers at the right time will significantly affect the quality of service (QoS) of the communication provided. To address this problem, this work proposes a graph-based customizable handover framework that considers both the handover timing and target while selecting a handover sequence that maintains QoS. A time-based graph is designed where the vertices represent the satellites' instances over a certain period of time and the edges' weights are the customizable handover criteria (i.e., data rate and delay in this work). The appropriate sequence and timing of handovers that fulfill the required QoS are obtained by finding the shortest path in the graph. Discussion and simulations, which were conducted on the Starlink Phase I constellation, show the low complexity and performance advantages of the proposed handover framework.