OpenRaiser/Intern-Atlas huggingface dataset:
Benchmarking Safety Monitors for Image Classifiers with Machine Learning
Venue: arXiv cs.AI 2021
Year: 2021

High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. Thus, their deployment in safety-critical applications such as autonomous vehicles is still an open issue. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state despite errors of the ML classifier. As the prediction from the ML is the core information directly impacting safety, many works are focusing on monitoring the ML model itself. Checking the efficiency of such monitors in the context of safety-critical applications is thus a significant challenge. Therefore, this paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers. Furthermore, we propose a framework covering the entire pipeline, from data generation to evaluation. Our approach measures monitor performance with a broader set of metrics than usually proposed in the literature. Moreover, we benchmark three different monitor approaches in 79 benchmark datasets containing five categories of out-of-distribution data for image classifiers: class novelty, noise, anomalies, distributional shifts, and adversarial attacks. Our results indicate that these monitors are no more accurate than a random monitor. We also release the code of all experiments for reproducibility.
Cybersickness-aware Tile-based Adaptive 360° Video Streaming
Venue: arXiv cs.NI 2021
Year: 2021

In contrast to traditional videos, the imaging in virtual reality (VR) is 360°, and it consumes larger bandwidth to transmit video contents. To reduce bandwidth consumption, tile-based streaming has been proposed to deliver the focused part of the video, instead of the whole one. On the other hand, the techniques to alleviate cybersickness, which is akin to motion sickness and happens when using digital displays, have not been jointly explored with the tile selection in VR. In this paper, we investigate Tile Selection with Cybersickness Control (TSCC) in an adaptive 360° video streaming system with cybersickness alleviation. We propose an m-competitive online algorithm with Cybersickness Indicator (CI) and Video Loss Indicator (VLI) to evaluate instant cybersickness and the total loss of video quality. Moreover, the algorithm exploits Sickness Migration Indicator (SMI) to evaluate the cybersickness accumulated over time and the increase of optical flow to improve the tile quality assignment. Simulations with a real network dataset show that our algorithm outperforms the baselines regarding video quality and cybersickness accumulation.
Behaviour-conditioned policies for cooperative reinforcement learning tasks
Venue: arXiv cs.LG 2021
Year: 2021

The cooperation among AI systems, and between AI systems and humans is becoming increasingly important. In various real-world tasks, an agent needs to cooperate with unknown partner agent types. This requires the agent to assess the behaviour of the partner agent during a cooperative task and to adjust its own policy to support the cooperation. Deep reinforcement learning models can be trained to deliver the required functionality but are known to suffer from sample inefficiency and slow learning. However, adapting to a partner agent behaviour during the ongoing task requires ability to assess the partner agent type quickly. We suggest a method, where we synthetically produce populations of agents with different behavioural patterns together with ground truth data of their behaviour, and use this data for training a meta-learner. We additionally suggest an agent architecture, which can efficiently use the generated data and gain the meta-learning capability. When an agent is equipped with such a meta-learner, it is capable of quickly adapting to cooperation with unknown partner agent types in new situations. This method can be used to automatically form a task distribution for meta-training from emerging behaviours that arise, for example, through self-play.
Prolog as a Querying Language for MongoDB
Venue: arXiv cs.DB 2021
Year: 2021

Today's database systems have shown to be capable of supporting AI applications that demand a lot of data processing. To this end, these systems incorporate powerful querying languages that go far beyond the mere retrieval of data, and provide sophisticated facilities for data processing as well. In the case of SQL, the language has been even demonstrated to be Turing-complete in some implementations of the language. In the area of NoSQL databases, a widely adopted one nowadays is the MongoDB database. Queries in MongoDB databases are represented as sequential stages within an aggregation pipeline where each stage defines a transformation of the input data, and passes the transformed data to the next stage. But aggregation queries tend to become rather large for more complex computational problems, lack organization into re-usable pieces, and are thus hard to debug and maintain. We propose a new database querying language called Mongolog which is syntactically a subset of the Prolog language, and we define its operational semantics through translations into aggregation pipelines. To this end, we make use of and extend the formal framework of the MQuery language which characterizes the aggregation framework set-theoretically.
Automated Aerial Animal Detection When Spatial Resolution Conditions Are Varied
Venue: arXiv cs.CV 2021
Year: 2021

Knowing where livestock are located enables optimized management and mustering. However, Australian farms are large meaning that many of Australia's livestock are unmonitored which impacts farm profit, animal welfare and the environment. Effective animal localisation and counting by analysing satellite imagery overcomes this management hurdle however, high resolution satellite imagery is expensive. Thus, to minimise cost the lowest spatial resolution data that enables accurate livestock detection should be selected. In our work, we determine the association between object detector performance and spatial degradation for cattle, sheep and dogs. Accurate ground truth was established using high resolution drone images which were then downsampled to various ground sample distances (GSDs). Both circular and cassegrain aperture optics were simulated to generate point spread functions (PSFs) corresponding to various optical qualities. By simulating the PSF, rather than approximating it as a Gaussian, the images were accurately degraded to match the spatial resolution and blurring structure of satellite imagery. Two existing datasets were combined and used to train and test a YoloV5 object detection network. Detector performance was found to drop steeply around a GSD of 0.5m/px and was associated with PSF matrix structure within this GSD region. Detector mAP performance fell by 52 percent when a cassegrain, rather than circular, aperture was used at a 0.5m/px GSD. Overall blurring magnitude also had a small impact when matched to GSD, as did the internal network resolution. Our results here inform the selection of remote sensing data requirements for animal detection tasks, allowing farmers and ecologists to use more accessible medium resolution imagery with confidence.
Context-Aware Unsupervised Clustering for Person Search
Venue: arXiv cs.CV 2021
Year: 2021

The existing person search methods use the annotated labels of person identities to train deep networks in a supervised manner that requires a huge amount of time and effort for human labeling. In this paper, we first introduce a novel framework of person search that is able to train the network in the absence of the person identity labels, and propose efficient unsupervised clustering methods to substitute the supervision process using annotated person identity labels. Specifically, we propose a hard negative mining scheme based on the uniqueness property that only a single person has the same identity to a given query person in each image. We also propose a hard positive mining scheme by using the contextual information of co-appearance that neighboring persons in one image tend to appear simultaneously in other images. The experimental results show that the proposed method achieves comparable performance to that of the state-of-the-art supervised person search methods, and furthermore outperforms the extended unsupervised person re-identification methods on the benchmark person search datasets.
Simulated annealing for optimization of graphs and sequences
Venue: arXiv cs.LG 2021
Year: 2021

Optimization of discrete structures aims at generating a new structure with the better property given an existing one, which is a fundamental problem in machine learning. Different from the continuous optimization, the realistic applications of discrete optimization (e.g., text generation) are very challenging due to the complex and long-range constraints, including both syntax and semantics, in discrete structures. In this work, we present SAGS, a novel Simulated Annealing framework for Graph and Sequence optimization. The key idea is to integrate powerful neural networks into metaheuristics (e.g., simulated annealing, SA) to restrict the search space in discrete optimization. We start by defining a sophisticated objective function, involving the property of interest and pre-defined constraints (e.g., grammar validity). SAGS searches from the discrete space towards this objective by performing a sequence of local edits, where deep generative neural networks propose the editing content and thus can control the quality of editing. We evaluate SAGS on paraphrase generation and molecule generation for sequence optimization and graph optimization, respectively. Extensive results show that our approach achieves state-of-the-art performance compared with existing paraphrase generation methods in terms of both automatic and human evaluations. Further, SAGS also significantly outperforms all the previous methods in molecule generation.
Development of an autism screening classification model for toddlers
Venue: arXiv cs.LG 2021
Year: 2021

Autism spectrum disorder ASD is a neurodevelopmental disorder associated with challenges in communication, social interaction, and repetitive behaviors. Getting a clear diagnosis for a child is necessary for starting early intervention and having access to therapy services. However, there are many barriers that hinder the screening of these kids for autism at an early stage which might delay further the access to therapeutic interventions. One promising direction for improving the efficiency and accuracy of ASD detection in toddlers is the use of machine learning techniques to build classifiers that serve the purpose. This paper contributes to this area and uses the data developed by Dr. Fadi Fayez Thabtah to train and test various machine learning classifiers for the early ASD screening. Based on various attributes, three models have been trained and compared which are Decision tree C4.5, Random Forest, and Neural Network. The three models provided very good accuracies based on testing data, however, it is the Neural Network that outperformed the other two models. This work contributes to the early screening of toddlers by helping identify those who have ASD traits and should pursue formal clinical diagnosis.
Observing a Moving Target -- Reliable Transmission of Debug Logs from Embedded Mobile Devices
Venue: arXiv cs.DC 2021
Year: 2021

Mobile embedded devices of the Internet of Things (IoT) face tight resource constraints and uncertain environments, including energy scarcity and unstable connectivity. This aggravates debugging, optimization, monitoring, etc.; for which logging information must be accessible throughout all phases of development and product life cycles. This work compares approaches for transmitting logs with regard to application requirements (e.g., bandwidth), resource consumption (e.g., memory), operating constraints (e.g., power supply), and the medium (e.g., UART, WiFi). A qualitative comparison suggests that the adequacy of approaches depends on the concrete application and the phase in the life cycle. We report from our case study, where the embedded mobile device is represented by a self-driving slot car (Carrera D132). With this target device, failure logs, new firmware, and monitoring data need to be exchanged. The gathered experiences support our qualitative discussion: Wireless techniques can suit the needs of many phases in the life cycle but it is particularly evident that energy consumption is crucial. With a loaded wireless stack and while transmitting logs, the car's operating voltage drop within a \~ 20 ms power interruption is \~ 1.6 times higher. This limits communication flexibility significantly.
Asynchronous Upper Confidence Bound Algorithms for Federated Linear Bandits
Venue: arXiv cs.LG 2021
Year: 2021

Linear contextual bandit is a popular online learning problem. It has been mostly studied in centralized learning settings. With the surging demand of large-scale decentralized model learning, e.g., federated learning, how to retain regret minimization while reducing communication cost becomes an open challenge. In this paper, we study linear contextual bandit in a federated learning setting. We propose a general framework with asynchronous model update and communication for a collection of homogeneous clients and heterogeneous clients, respectively. Rigorous theoretical analysis is provided about the regret and communication cost under this distributed learning framework; and extensive empirical evaluations demonstrate the effectiveness of our solution.
Realizing Forward Defense in the Cyber Domain
Venue: arXiv cs.CR 2021
Year: 2021

With the recognition of cyberspace as an operating domain, concerted effort is now being placed on addressing it in the whole-of-domain manner found in land, sea, undersea, air, and space domains. Among the first steps in this effort is applying the standard supporting concepts of security, defense, and deterrence to the cyber domain. This paper presents an architecture that helps realize forward defense in cyberspace, wherein adversarial actions are repulsed as close to the origin as possible. However, substantial work remains in making the architecture an operational reality including furthering fundamental research cyber science, conducting design trade-off analysis, and developing appropriate public policy frameworks.
Balanced Masked and Standard Face Recognition
Venue: arXiv cs.CV 2021
Year: 2021

We present the improved network architecture, data augmentation, and training strategies for the Webface track and Insightface/Glint360K track of the masked face recognition challenge of ICCV2021. One of the key goals is to have a balanced performance of masked and standard face recognition. In order to prevent the overfitting for the masked face recognition, we control the total number of masked faces by not more than 10\% of the total face recognition in the training dataset. We propose a few key changes to the face recognition network including a new stem unit, drop block, face detection and alignment using YOLO5Face, feature concatenation, a cycle cosine learning rate, etc. With this strategy, we achieve good and balanced performance for both masked and standard face recognition.
Differentiable Spline Approximations
Venue: arXiv cs.LG 2021
Year: 2021

The paradigm of differentiable programming has significantly enhanced the scope of machine learning via the judicious use of gradient-based optimization. However, standard differentiable programming methods (such as autodiff) typically require that the machine learning models be differentiable, limiting their applicability. Our goal in this paper is to use a new, principled approach to extend gradient-based optimization to functions well modeled by splines, which encompass a large family of piecewise polynomial models. We derive the form of the (weak) Jacobian of such functions and show that it exhibits a block-sparse structure that can be computed implicitly and efficiently. Overall, we show that leveraging this redesigned Jacobian in the form of a differentiable "layer" in predictive models leads to improved performance in diverse applications such as image segmentation, 3D point cloud reconstruction, and finite element analysis.
Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA
Venue: arXiv cs.CL 2021
Year: 2021

Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook's content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5's pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have "understood" the textbook (or perhaps misunderstood the questions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).
Effectiveness of Optimization Algorithms in Deep Image Classification
Venue: arXiv cs.CV 2021
Year: 2021

Adam is applied widely to train neural networks. Different kinds of Adam methods with different features pop out. Recently two new adam optimizers, AdaBelief and Padam are introduced among the community. We analyze these two adam optimizers and compare them with other conventional optimizers (Adam, SGD + Momentum) in the scenario of image classification. We evaluate the performance of these optimization algorithms on AlexNet and simplified versions of VGGNet, ResNet using the EMNIST dataset. (Benchmark algorithm is available at \hyperref[https://github.com/chuiyunjun/projectCSC413]{https://github.com/chuiyunjun/projectCSC413}).
Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answering
Venue: arXiv cs.IR 2021
Year: 2021

One key feature of dense passage retrievers (DPR) is the use of separate question and passage encoder in a bi-encoder design. Previous work on generalization of DPR mainly focus on testing both encoders in tandem on out-of-distribution (OOD) question-answering (QA) tasks, which is also known as domain adaptation. However, it is still unknown how DPR's individual question/passage encoder affects generalization. Specifically, in this paper, we want to know how an in-distribution (IND) question/passage encoder would generalize if paired with an OOD passage/question encoder from another domain. We refer to this challenge as \textit{encoder adaptation}. To answer this question, we inspect different combinations of DPR's question and passage encoder learned from five benchmark QA datasets on both in-domain and out-of-domain questions. We find that the passage encoder has more influence on the lower bound of generalization while the question encoder seems to affect the upper bound in general. For example, applying an OOD passage encoder usually hurts the retrieval accuracy while an OOD question encoder sometimes even improves the accuracy.
Geography of Science: Competitiveness and Inequality
Venue: arXiv cs.DL 2021
Year: 2021

Using ideas and tools of complexity science we design a holistic measure of \textit{Scientific Fitness}, encompassing the scientific knowledge, capabilities and competitiveness of a research system. We characterize the temporal dynamics of Scientific Fitness and R\&D expenditures at the geographical scale of nations, highlighting patterns of similar research systems, and showing how developing nations (China in particular) are quickly catching up the developed ones. Down-scaling the aggregation level of the analysis, we find that even developed nations show a considerable level of inequality in the Scientific Fitness of their internal regions. Further, we assess comparatively how the competitiveness of each geographic region is distributed over the spectrum of research sectors. Overall, the Scientific Fitness represents the first high quality estimation of the scientific strength of nations and regions, opening new policy-making applications for better allocating resources, filling inequality gaps and ultimately promoting innovation.
Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning
Venue: arXiv cs.CL 2021
Year: 2021

Privacy is important considering the financial Domain as such data is highly confidential and sensitive. Natural Language Processing (NLP) techniques can be applied for text classification and entity detection purposes in financial domains such as customer feedback sentiment analysis, invoice entity detection, categorisation of financial documents by type etc. Due to the sensitive nature of such data, privacy measures need to be taken for handling and training large models with such data. In this work, we propose a contextualized transformer (BERT and RoBERTa) based text classification model integrated with privacy features such as Differential Privacy (DP) and Federated Learning (FL). We present how to privately train NLP models and desirable privacy-utility tradeoffs and evaluate them on the Financial Phrase Bank dataset.
HDR-cGAN: Single LDR to HDR Image Translation using Conditional GAN
Venue: arXiv cs.CV 2021
Year: 2021

The prime goal of digital imaging techniques is to reproduce the realistic appearance of a scene. Low Dynamic Range (LDR) cameras are incapable of representing the wide dynamic range of the real-world scene. The captured images turn out to be either too dark (underexposed) or too bright (overexposed). Specifically, saturation in overexposed regions makes the task of reconstructing a High Dynamic Range (HDR) image from single LDR image challenging. In this paper, we propose a deep learning based approach to recover details in the saturated areas while reconstructing the HDR image. We formulate this problem as an image-to-image (I2I) translation task. To this end, we present a novel conditional GAN (cGAN) based framework trained in an end-to-end fashion over the HDR-REAL and HDR-SYNTH datasets. Our framework uses an overexposed mask obtained from a pre-trained segmentation model to facilitate the hallucination task of adding details in the saturated regions. We demonstrate the effectiveness of the proposed method by performing an extensive quantitative and qualitative comparison with several state-of-the-art single-image HDR reconstruction techniques.