OpenRaiser/Intern-Atlas huggingface dataset:
A theory of capacity and sparse neural encoding
Venue: arXiv cs.LG 2021
Year: 2021

Motivated by biological considerations, we study sparse neural maps from an input layer to a target layer with sparse activity, and specifically the problem of storing $K$ input-target associations $(x,y)$, or memories, when the target vectors $y$ are sparse. We mathematically prove that $K$ undergoes a phase transition and that in general, and somewhat paradoxically, sparsity in the target layers increases the storage capacity of the map. The target vectors can be chosen arbitrarily, including in random fashion, and the memories can be both encoded and decoded by networks trained using local learning rules, including the simple Hebb rule. These results are robust under a variety of statistical assumptions on the data. The proofs rely on elegant properties of random polytopes and sub-gaussian random vector variables. Open problems and connections to capacity theories and polynomial threshold maps are discussed.
Clarification of Video Retrieval Query Results by the Automated Insertion of Supporting Shots
Venue: arXiv cs.MM 2021
Year: 2021

Computational Video Editing Systems output video generally follows a particular form, e.g. conversation or music videos, in this way they are domain specific. We describe a recent development in our video annotation and segmentation system to support general computational video editing in which we derive a single generic editing strategy from general cinema narrative principles instead of using a hierarchical film gram-mar. We demonstrate how this single principle coupled with a database of scripts derived from annotated videos leverages the existing video editing knowledge encoded within the editing of those sequences in a flexible and generic manner. We discuss the cinema theory foundations for this generic editing strategy, review the algorithms used to effect it, and goon by means of examples to show its appropriateness in an automated system.
Machine Translation Customization via Automatic Training Data Selection from the Web
Venue: arXiv cs.CL 2021
Year: 2021

Machine translation (MT) systems, especially when designed for an industrial setting, are trained with general parallel data derived from the Web. Thus, their style is typically driven by word/structure distribution coming from the average of many domains. In contrast, MT customers want translations to be specialized to their domain, for which they are typically able to provide text samples. We describe an approach for customizing MT systems on specific domains by selecting data similar to the target customer data to train neural translation models. We build document classifiers using monolingual target data, e.g., provided by the customers to select parallel training data from Web crawled data. Finally, we train MT models on our automatically selected data, obtaining a system specialized to the target domain. We tested our approach on the benchmark from WMT-18 Translation Task for News domains enabling comparisons with state-of-the-art MT systems. The results show that our models outperform the top systems while using less data and smaller models.
On a notion of independence proposed by Teddy Seidenfeld
Venue: arXiv cs.AI 2021
Year: 2021

Teddy Seidenfeld has been arguing for quite a long time that binary preference models are not powerful enough to deal with a number of crucial aspects of imprecision and indeterminacy in uncertain inference and decision making. It is at his insistence that we initiated our study of so-called sets of desirable option sets, which we have argued elsewhere provides an elegant and powerful approach to dealing with general, binary as well as non-binary, decision-making under uncertainty. We use this approach here to explore an interesting notion of irrelevance (and independence), first suggested by Seidenfeld in an example intended as a criticism of a number of specific decision methodologies based on (convex) binary preferences. We show that the consequences of making such an irrelevance or independence assessment are very strong, and might be used to argue for the use of so-called mixing choice functions, and E-admissibility as the resulting decision scheme.
EMDS-5: Environmental Microorganism Image Dataset Fifth Version for Multiple Image Analysis Tasks
Venue: arXiv cs.CV 2021
Year: 2021

Environmental Microorganism Data Set Fifth Version (EMDS-5) is a microscopic image dataset including original Environmental Microorganism (EM) images and two sets of Ground Truth (GT) images. The GT image sets include a single-object GT image set and a multi-object GT image set. The EMDS-5 dataset has 21 types of EMs, each of which contains 20 original EM images, 20 single-object GT images and 20 multi-object GT images. EMDS-5 can realize to evaluate image preprocessing, image segmentation, feature extraction, image classification and image retrieval functions. In order to prove the effectiveness of EMDS-5, for each function, we select the most representative algorithms and price indicators for testing and evaluation. The image preprocessing functions contain two parts: image denoising and image edge detection. Image denoising uses nine kinds of filters to denoise 13 kinds of noises, respectively. In the aspect of edge detection, six edge detection operators are used to detect the edges of the images, and two evaluation indicators, peak-signal to noise ratio and mean structural similarity, are used for evaluation. Image segmentation includes single-object image segmentation and multi-object image segmentation. Six methods are used for single-object image segmentation, while k-means and U-net are used for multi-object segmentation.We extract nine features from the images in EMDS-5 and use the Support Vector Machine classifier for testing. In terms of image classification, we select the VGG16 feature to test different classifiers. We test two types of retrieval approaches: texture feature retrieval and deep learning feature retrieval. We select the last layer of features of these two deep learning networks as feature vectors. We use mean average precision as the evaluation index for retrieval.
GLAM: Graph Learning by Modeling Affinity to Labeled Nodes for Graph Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Graph Neural Networks have shown excellent performance on semi-supervised classification tasks. However, they assume access to a graph that may not be often available in practice. In the absence of any graph, constructing k-Nearest Neighbor (kNN) graphs from the given data have shown to give improvements when used with GNNs over other semi-supervised methods. This paper proposes a semi-supervised graph learning method for cases when there are no graphs available. This method learns a graph as a convex combination of the unsupervised kNN graph and a supervised label-affinity graph. The label-affinity graph directly captures all the nodes' label-affinity with the labeled nodes, i.e., how likely a node has the same label as the labeled nodes. This affinity measure contrasts with the kNN graph where the metric measures closeness in the feature space. Our experiments suggest that this approach gives close to or better performance (up to 1.5%), while being simpler and faster (up to 70x) to train, than state-of-the-art graph learning methods. We also conduct several experiments to highlight the importance of individual components and contrast them with state-of-the-art methods.
MHDeep: Mental Health Disorder Detection System based on Body-Area and Deep Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Mental health problems impact quality of life of millions of people around the world. However, diagnosis of mental health disorders is a challenging problem that often relies on self-reporting by patients about their behavioral patterns. Therefore, there is a need for new strategies for diagnosis of mental health problems. The recent introduction of body-area networks consisting of a plethora of accurate sensors embedded in smartwatches and smartphones and deep neural networks (DNNs) points towards a possible solution. However, disease diagnosis based on WMSs and DNNs, and their deployment on edge devices, remains a challenging problem. To this end, we propose a framework called MHDeep that utilizes commercially available WMSs and efficient DNN models to diagnose three important mental health disorders: schizoaffective, major depressive, and bipolar. MHDeep uses eight different categories of data obtained from sensors integrated in a smartwatch and smartphone. Due to limited available data, MHDeep uses a synthetic data generation module to augment real data with synthetic data drawn from the same probability distribution. We use the synthetic dataset to pre-train the DNN models, thus imposing a prior on the weights. We use a grow-and-prune DNN synthesis approach to learn both the architecture and weights during the training process. We use three different data partitions to evaluate the MHDeep models trained with data collected from 74 individuals. We conduct data instance level and patient level evaluations. MHDeep achieves an average test accuracy of 90.4%, 87.3%, and 82.4%, respectively, for classifications between healthy instances and schizoaffective disorder instances, major depressive disorder instances, and bipolar disorder instances. At the patient level, MHDeep DNNs achieve an accuracy of 100%, 100%, and 90.0% for the three mental health disorders, respectively.
Understanding and Enhancing the Use of Context for Machine Translation
Venue: arXiv cs.CL 2021
Year: 2021

To understand and infer meaning in language, neural models have to learn complicated nuances. Discovering distinctive linguistic phenomena from data is not an easy task. For instance, lexical ambiguity is a fundamental feature of language which is challenging to learn. Even more prominently, inferring the meaning of rare and unseen lexical units is difficult with neural networks. Meaning is often determined from context. With context, languages allow meaning to be conveyed even when the specific words used are not known by the reader. To model this learning process, a system has to learn from a few instances in context and be able to generalize well to unseen cases. The learning process is hindered when training data is scarce for a task. Even with sufficient data, learning patterns for the long tail of the lexical distribution is challenging. In this thesis, we focus on understanding certain potentials of contexts in neural models and design augmentation models to benefit from them. We focus on machine translation as an important instance of the more general language understanding problem. To translate from a source language to a target language, a neural model has to understand the meaning of constituents in the provided context and generate constituents with the same meanings in the target language. This task accentuates the value of capturing nuances of language and the necessity of generalization from few observations. The main problem we study in this thesis is what neural machine translation models learn from data and how we can devise more focused contexts to enhance this learning. Looking more in-depth into the role of context and the impact of data on learning models is essential to advance the NLP field. Moreover, it helps highlight the vulnerabilities of current neural networks and provides insights into designing more robust models.
CheXseg: Combining Expert Annotations with DNN-generated Saliency Maps for X-ray Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

Medical image segmentation models are typically supervised by expert annotations at the pixel-level, which can be expensive to acquire. In this work, we propose a method that combines the high quality of pixel-level expert annotations with the scale of coarse DNN-generated saliency maps for training multi-label semantic segmentation models. We demonstrate the application of our semi-supervised method, which we call CheXseg, on multi-label chest X-ray interpretation. We find that CheXseg improves upon the performance (mIoU) of fully-supervised methods that use only pixel-level expert annotations by 9.7% and weakly-supervised methods that use only DNN-generated saliency maps by 73.1%. Our best method is able to match radiologist agreement on three out of ten pathologies and reduces the overall performance gap by 57.2% as compared to weakly-supervised methods.
A Hierarchical Conditional Random Field-based Attention Mechanism Approach for Gastric Histopathology Image Classification
Venue: arXiv cs.CV 2021
Year: 2021

In the Gastric Histopathology Image Classification (GHIC) tasks, which are usually weakly supervised learning missions, there is inevitably redundant information in the images. Therefore, designing networks that can focus on effective distinguishing features has become a popular research topic. In this paper, to accomplish the tasks of GHIC superiorly and to assist pathologists in clinical diagnosis, an intelligent Hierarchical Conditional Random Field based Attention Mechanism (HCRF-AM) model is proposed. The HCRF-AM model consists of an Attention Mechanism (AM) module and an Image Classification (IC) module. In the AM module, an HCRF model is built to extract attention regions. In the IC module, a Convolutional Neural Network (CNN) model is trained with the attention regions selected and then an algorithm called Classification Probability-based Ensemble Learning is applied to obtain the image-level results from patch-level output of the CNN. In the experiment, a classification specificity of 96.67% is achieved on a gastric histopathology dataset with 700 images. Our HCRF-AM model demonstrates high classification performance and shows its effectiveness and future potential in the GHIC field.
The Effects of Image Distribution and Task on Adversarial Robustness
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we propose an adaptation to the area under the curve (AUC) metric to measure the adversarial robustness of a model over a particular $ε$-interval $[ε_0, ε_1]$ (interval of adversarial perturbation strengths) that facilitates unbiased comparisons across models when they have different initial $ε_0$ performance. This can be used to determine how adversarially robust a model is to different image distributions or task (or some other variable); and/or to measure how robust a model is comparatively to other models. We used this adversarial robustness metric on models of an MNIST, CIFAR-10, and a Fusion dataset (CIFAR-10 + MNIST) where trained models performed either a digit or object recognition task using a LeNet, ResNet50, or a fully connected network (FullyConnectedNet) architecture and found the following: 1) CIFAR-10 models are inherently less adversarially robust than MNIST models; 2) Both the image distribution and task that a model is trained on can affect the adversarial robustness of the resultant model. 3) Pretraining with a different image distribution and task sometimes carries over the adversarial robustness induced by that image distribution and task in the resultant model; Collectively, our results imply non-trivial differences of the learned representation space of one perceptual system over another given its exposure to different image statistics or tasks (mainly objects vs digits). Moreover, these results hold even when model systems are equalized to have the same level of performance, or when exposed to approximately matched image statistics of fusion images but with different tasks.
Tractable Computation of Expected Kernels
Venue: arXiv cs.LG 2021
Year: 2021

Computing the expectation of kernel functions is a ubiquitous task in machine learning, with applications from classical support vector machines to exploiting kernel embeddings of distributions in probabilistic modeling, statistical inference, causal discovery, and deep learning. In all these scenarios, we tend to resort to Monte Carlo estimates as expectations of kernels are intractable in general. In this work, we characterize the conditions under which we can compute expected kernels exactly and efficiently, by leveraging recent advances in probabilistic circuit representations. We first construct a circuit representation for kernels and propose an approach to such tractable computation. We then demonstrate possible advancements for kernel embedding frameworks by exploiting tractable expected kernels to derive new algorithms for two challenging scenarios: 1) reasoning under missing data with kernel support vector regressors; 2) devising a collapsed black-box importance sampling scheme. Finally, we empirically evaluate both algorithms and show that they outperform standard baselines on a variety of datasets.
CFLMEC: Cooperative Federated Learning for Mobile Edge Computing
Venue: arXiv cs.NI 2021
Year: 2021

We investigate a cooperative federated learning framework among devices for mobile edge computing, named CFLMEC, where devices co-exist in a shared spectrum with interference. Keeping in view the time-average network throughput of cooperative federated learning framework and spectrum scarcity, we focus on maximize the admission data to the edge server or the near devices, which fills the gap of communication resource allocation for devices with federated learning. In CFLMEC, devices can transmit local models to the corresponding devices or the edge server in a relay race manner, and we use a decomposition approach to solve the resource optimization problem by considering maximum data rate on sub-channel, channel reuse and wireless resource allocation in which establishes a primal-dual learning framework and batch gradient decent to learn the dynamic network with outdated information and predict the sub-channel condition. With aim at maximizing throughput of devices, we propose communication resource allocation algorithms with and without sufficient sub-channels for strong reliance on edge servers (SRs) in cellular link, and interference aware communication resource allocation algorithm for less reliance on edge servers (LRs) in D2D link. Extensive simulation results demonstrate the CFLMEC can achieve the highest throughput of local devices comparing with existing works, meanwhile limiting the number of the sub-channels.
Web-based Application for Detecting Indonesian Clickbait Headlines using IndoBERT
Venue: arXiv cs.CL 2021
Year: 2021

With increasing usage of clickbaits in Indonesian Online News, newsworthy articles sometimes get buried among clickbaity news. A reliable and lightweight tool is needed to detect such clickbaits on-the-go. Leveraging state-of-the-art natural language processing model BERT, a RESTful API based application is developed. This study offloaded the computing resources needed to train the model on the cloud server, while the client-side application only needs to send a request to the API and the cloud server will handle the rest. This study proposed the design and developed a web-based application to detect clickbait in Indonesian using IndoBERT as a language model. The application usage is discussed and available for public use with a performance of mean ROC-AUC of 89%.
A Ransomware Classification Framework Based on File-Deletion and File-Encryption Attack Structures
Venue: arXiv cs.CR 2021
Year: 2021

Ransomware has emerged as an infamous malware that has not escaped a lot of myths and inaccuracies from media hype. Victims are not sure whether or not to pay a ransom demand without fully understanding the lurking consequences. In this paper, we present a ransomware classification framework based on file-deletion and file-encryption attack structures that provides a deeper comprehension of potential flaws and inadequacies exhibited in ransomware. We formulate a threat and attack model representative of a typical ransomware attack process from which we derive the ransomware categorization framework based on a proposed classification algorithm. The framework classifies the virulence of a ransomware attack to entail the overall effectiveness of potential ways of recovering the attacked data without paying the ransom demand as well as the technical prowess of the underlying attack structures. Results of the categorization, in increasing severity from CAT1 through to CAT5, show that many ransomwares exhibit flaws in their implementation of encryption and deletion attack structures which make data recovery possible without paying the ransom. The most severe categories CAT4 and CAT5 are better mitigated by exploiting encryption essentials while CAT3 can be effectively mitigated via reverse engineering. CAT1 and CAT2 are not common and are easily mitigated without any decryption essentials.
Dimensions of Timescales in Neuromorphic Computing Systems
Venue: arXiv cs.ET 2021
Year: 2021

This article is a public deliverable of the EU project "Memory technologies with multi-scale time constants for neuromorphic architectures" (MeMScales, https://memscales.eu, Call ICT-06-2019 Unconventional Nanoelectronics, project number 871371). This arXiv version is a verbatim copy of the deliverable report, with administrative information stripped. It collects a wide and varied assortment of phenomena, models, research themes and algorithmic techniques that are connected with timescale phenomena in the fields of computational neuroscience, mathematics, machine learning and computer science, with a bias toward aspects that are relevant for neuromorphic engineering. It turns out that this theme is very rich indeed and spreads out in many directions which defy a unified treatment. We collected several dozens of sub-themes, each of which has been investigated in specialized settings (in the neurosciences, mathematics, computer science and machine learning) and has been documented in its own body of literature. The more we dived into this diversity, the more it became clear that our first effort to compose a survey must remain sketchy and partial. We conclude with a list of insights distilled from this survey which give general guidelines for the design of future neuromorphic systems.
An agile and distributed mechanism for inter-domain network slicing in next generation mobile networks
Venue: arXiv cs.GT 2021
Year: 2021

Network slicing is emerging as a promising method to provide sought-after versatility and flexibility to cope with ever-increasing demands. To realize such potential advantages and to meet the challenging requirements of various network slices in an on-demand fashion, we need to develop an agile and distributed mechanism for resource provisioning to different network slices in a heterogeneous multi-resource multi-domain mobile network environment. We formulate inter-domain resource provisioning to network slices in such an environment as an optimization problem which maximizes social welfare among network slice tenants (so that maximizing tenants' satisfaction), while minimizing operational expenditures for infrastructure service providers at the same time. To solve the envisioned problem, we implement an iterative auction game among network slice tenants, on one hand, and a plurality of price-taking subnet service providers, on the other hand. We show that the proposed solution method results in a distributed privacy-saving mechanism which converges to the optimal solution of the described optimization problem. In addition to providing analytical results to characterize the performance of the proposed mechanism, we also employ numerical evaluations to validate the results, demonstrate convergence of the presented algorithm, and show the enhanced performance of the proposed approach (in terms of resource utilization, fairness and operational costs) against the existing solutions.