OpenRaiser/Intern-Atlas huggingface dataset:
Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks
Venue: arXiv cs.CL 2022
Year: 2022

We introduce Dynatask: an open source system for setting up custom NLP tasks that aims to greatly lower the technical knowledge and effort required for hosting and evaluating state-of-the-art NLP models, as well as for conducting model in the loop data collection with crowdworkers. Dynatask is integrated with Dynabench, a research platform for rethinking benchmarking in AI that facilitates human and model in the loop data collection and evaluation. To create a task, users only need to write a short task configuration file from which the relevant web interfaces and model hosting infrastructure are automatically generated. The system is available at https://dynabench.org/ and the full library can be found at https://github.com/facebookresearch/dynabench.
An Exploration of Active Learning for Affective Digital Phenotyping
Venue: arXiv cs.LG 2022
Year: 2022

Some of the most severe bottlenecks preventing widespread development of machine learning models for human behavior include a dearth of labeled training data and difficulty of acquiring high quality labels. Active learning is a paradigm for using algorithms to computationally select a useful subset of data points to label using metrics for model uncertainty and data similarity. We explore active learning for naturalistic computer vision emotion data, a particularly heterogeneous and complex data space due to inherently subjective labels. Using frames collected from gameplay acquired from a therapeutic smartphone game for children with autism, we run a simulation of active learning using gameplay prompts as metadata to aid in the active learning process. We find that active learning using information generated during gameplay slightly outperforms random selection of the same number of labeled frames. We next investigate a method to conduct active learning with subjective data, such as in affective computing, and where multiple crowdsourced labels can be acquired for each image. Using the Child Affective Facial Expression (CAFE) dataset, we simulate an active learning process for crowdsourcing many labels and find that prioritizing frames using the entropy of the crowdsourced label distribution results in lower categorical cross-entropy loss compared to random frame selection. Collectively, these results demonstrate pilot evaluations of two novel active learning approaches for subjective affective data collected in noisy settings.
Digital Twin Virtualization with Machine Learning for IoT and Beyond 5G Networks: Research Directions for Security and Optimal Control
Venue: arXiv cs.NI 2022
Year: 2022

Digital twin (DT) technologies have emerged as a solution for real-time data-driven modeling of cyber physical systems (CPS) using the vast amount of data available by Internet of Things (IoT) networks. In this position paper, we elucidate unique characteristics and capabilities of a DT framework that enables realization of such promises as online learning of a physical environment, real-time monitoring of assets, Monte Carlo heuristic search for predictive prevention, on-policy, and off-policy reinforcement learning in real-time. We establish a conceptual layered architecture for a DT framework with decentralized implementation on cloud computing and enabled by artificial intelligence (AI) services for modeling, event detection, and decision-making processes. The DT framework separates the control functions, deployed as a system of logically centralized process, from the physical devices under control, much like software-defined networking (SDN) in fifth generation (5G) wireless networks. We discuss the moment of the DT framework in facilitating implementation of network-based control processes and its implications for critical infrastructure. To clarify the significance of DT in lowering the risk of development and deployment of innovative technologies on existing system, we discuss the application of implementing zero trust architecture (ZTA) as a necessary security framework in future data-driven communication networks.
Towards On-Board Panoptic Segmentation of Multispectral Satellite Images
Venue: arXiv cs.CV 2022
Year: 2022

With tremendous advancements in low-power embedded computing devices and remote sensing instruments, the traditional satellite image processing pipeline which includes an expensive data transfer step prior to processing data on the ground is being replaced by on-board processing of captured data. This paradigm shift enables critical and time-sensitive analytic intelligence to be acquired in a timely manner on-board the satellite itself. However, at present, the on-board processing of multi-spectral satellite images is limited to classification and segmentation tasks. Extending this processing to its next logical level, in this paper we propose a lightweight pipeline for on-board panoptic segmentation of multi-spectral satellite images. Panoptic segmentation offers major economic and environmental insights, ranging from yield estimation from agricultural lands to intelligence for complex military applications. Nevertheless, the on-board intelligence extraction raises several challenges due to the loss of temporal observations and the need to generate predictions from a single image sample. To address this challenge, we propose a multimodal teacher network based on a cross-modality attention-based fusion strategy to improve the segmentation accuracy by exploiting data from multiple modes. We also propose an online knowledge distillation framework to transfer the knowledge learned by this multi-modal teacher network to a uni-modal student which receives only a single frame input, and is more appropriate for an on-board environment. We benchmark our approach against existing state-of-the-art panoptic segmentation models using the PASTIS multi-spectral panoptic segmentation dataset considering an on-board processing setting. Our evaluations demonstrate a substantial increase in accuracy metrics compared to the existing state-of-the-art models.
Region Rebalance for Long-Tailed Semantic Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we study the problem of class imbalance in semantic segmentation. We first investigate and identify the main challenges of addressing this issue through pixel rebalance. Then a simple and yet effective region rebalance scheme is derived based on our analysis. In our solution, pixel features belonging to the same class are grouped into region features, and a rebalanced region classifier is applied via an auxiliary region rebalance branch during training. To verify the flexibility and effectiveness of our method, we apply the region rebalance module into various semantic segmentation methods, such as Deeplabv3+, OCRNet, and Swin. Our strategy achieves consistent improvement on the challenging ADE20K and COCO-Stuff benchmark. In particular, with the proposed region rebalance scheme, state-of-the-art BEiT receives +0.7% gain in terms of mIoU on the ADE20K val set.
Streaming Algorithms for Multitasking Scheduling with Shared Processing
Venue: arXiv cs.DS 2022
Year: 2022

In this paper, we design the first streaming algorithms for the problem of multitasking scheduling on parallel machines with shared processing. In one pass, our streaming approximation schemes can provide an approximate value of the optimal makespan. If the jobs can be read in two passes, the algorithm can find the schedule with the approximate value. This work not only provides an algorithmic big data solution for the studied problem, but also gives an insight into the design of streaming algorithms for other problems in the area of scheduling.
Compute- and Data-Intensive Networks: The Key to the Metaverse
Venue: arXiv cs.NI 2022
Year: 2022

The worlds of computing, communication, and storage have for a long time been treated separately, and even the recent trends of cloud computing, distributed computing, and mobile edge computing have not fundamentally changed the role of networks, still designed to move data between end users and pre-determined computation nodes, without true optimization of the end-to-end compute-communication process. However, the emergence of Metaverse applications, where users consume multimedia experiences that result from the real-time combination of distributed live sources and stored digital assets, has changed the requirements for, and possibilities of, systems that provide distributed caching, computation, and communication. We argue that the real-time interactive nature and high demands on data storage, streaming rates, and processing power of Metaverse applications will accelerate the merging of the cloud into the network, leading to highly-distributed tightly-integrated compute- and data-intensive networks becoming universal compute platforms for next-generation digital experiences. In this paper, we first describe the requirements of Metaverse applications and associated supporting infrastructure, including relevant use cases. We then outline a comprehensive cloud network flow mathematical framework, designed for the end-to-end optimization and control of such systems, and show numerical results illustrating its promising role for the efficient operation of Metaverse-ready networks.
Fact Checking with Insufficient Evidence
Venue: arXiv cs.CL 2022
Year: 2022

Automating the fact checking (FC) process relies on information obtained from external sources. In this work, we posit that it is crucial for FC models to make veracity predictions only when there is sufficient evidence and otherwise indicate when it is not enough. To this end, we are the first to study what information FC models consider sufficient by introducing a novel task and advancing it with three main contributions. First, we conduct an in-depth empirical analysis of the task with a new fluency-preserving method for omitting information from the evidence at the constituent and sentence level. We identify when models consider the remaining evidence (in)sufficient for FC, based on three trained models with different Transformer architectures and three FC datasets. Second, we ask annotators whether the omitted evidence was important for FC, resulting in a novel diagnostic dataset, SufficientFacts, for FC with omitted evidence. We find that models are least successful in detecting missing evidence when adverbial modifiers are omitted (21% accuracy), whereas it is easiest for omitted date modifiers (63% accuracy). Finally, we propose a novel data augmentation strategy for contrastive self-learning of missing evidence by employing the proposed omission method combined with tri-training. It improves performance for Evidence Sufficiency Prediction by up to 17.8 F1 score, which in turn improves FC performance by up to 2.6 F1 score.
Non-Linear Speech coding with MLP, RBF and Elman based prediction
Venue: arXiv cs.SD 2022
Year: 2022

In this paper we propose a nonlinear scalar predictor based on a combination of Multi Layer Perceptron, Radial Basis Functions and Elman networks. This system is applied to speech coding in an ADPCM backward scheme. The combination of this predictors improves the results of one predictor alone. A comparative study of this three neural networks for speech prediction is also presented.
Improved and Efficient Conversational Slot Labeling through Question Answering
Venue: arXiv cs.CL 2022
Year: 2022

Transformer-based pretrained language models (PLMs) offer unmatched performance across the majority of natural language understanding (NLU) tasks, including a body of question answering (QA) tasks. We hypothesize that improvements in QA methodology can also be directly exploited in dialog NLU; however, dialog tasks must be \textit{reformatted} into QA tasks. In particular, we focus on modeling and studying \textit{slot labeling} (SL), a crucial component of NLU for dialog, through the QA optics, aiming to improve both its performance and efficiency, and make it more effective and resilient to working with limited task data. To this end, we make a series of contributions: 1) We demonstrate how QA-tuned PLMs can be applied to the SL task, reaching new state-of-the-art performance, with large gains especially pronounced in such low-data regimes. 2) We propose to leverage contextual information, required to tackle ambiguous values, simply through natural language. 3) Efficiency and compactness of QA-oriented fine-tuning are boosted through the use of lightweight yet effective adapter modules. 4) Trading-off some of the quality of QA datasets for their size, we experiment with larger automatically generated QA datasets for QA-tuning, arriving at even higher performance. Finally, our analysis suggests that our novel QA-based slot labeling models, supported by the PLMs, reach a performance ceiling in high-data regimes, calling for more challenging and more nuanced benchmarks in future work.
Gradient Descent Bit-Flipping Decoding with Momentum
Venue: arXiv cs.IT 2022
Year: 2022

In this paper, we propose a Gradient Descent Bit-Flipping (GDBF) decoding with momentum, which considers past updates to provide inertia to the decoding process. We show that GDBF or randomized GDBF decoders with momentum may closely approach the floating-point Belief-Propagation decoding performance, and even outperform it in the error-floor region, especially for graphs with high connectivity degree.
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
Venue: arXiv cs.HC 2022
Year: 2022

We use a deep learning based approach to predict whether a selected element in a mobile UI screenshot will be perceived by users as tappable, based on pixels only instead of view hierarchies required by previous work. To help designers better understand model predictions and to provide more actionable design feedback than predictions alone, we additionally use ML interpretability techniques to help explain the output of our model. We use XRAI to highlight areas in the input screenshot that most strongly influence the tappability prediction for the selected region, and use k-Nearest Neighbors to present the most similar mobile UIs from the dataset with opposing influences on tappability perception.
Less is More: Fairness in Wide-Area Proof-of-Work Blockchain Networks
Venue: arXiv cs.NI 2022
Year: 2022

Blockchain is rapidly emerging as an important class of network application, with a unique set of trust, security and transparency properties. In a blockchain system, participants record and update the `server-side' state of an application as blocks of a replicated, immutable ledger using a consensus protocol over the Internet. Mining blocks has become lucrative in recent years; e.g., a miner receives over USD 200,000 per mined block in Bitcoin today. A key factor affecting mining rewards, is the latency of broadcasting blocks over the network. In this paper, we consider the problem of topology design for optimizing mining rewards in a wide-area blockchain network that uses a Proof-of-Work protocol for consensus. Contrary to general wisdom that a faster network is always better for miners, we show a counter intuitive result where a slower network is actually beneficial to some miners. This is because competing miners must choose neighbors that not only decrease their own latency to others, but also ensure that the latency between other miners do not decrease because of itself. We formalize this problem, and provide both theoretical analysis and experimental results to support our claim.
Quadratic Approximation Manifold for Mitigating the Kolmogorov Barrier in Nonlinear Projection-Based Model Order Reduction
Venue: arXiv cs.CE 2022
Year: 2022

A quadratic approximation manifold is presented for performing nonlinear, projection-based, model order reduction (PMOR). It constitutes a departure from the traditional affine subspace approximation that is aimed at mitigating the Kolmogorov barrier for nonlinear PMOR, particularly for convection-dominated transport problems. It builds on the data-driven approach underlying the traditional construction of projection-based reduced-order models (PROMs); is application-independent; is linearization-free; and therefore is robust for highly nonlinear problems. Most importantly, this approximation leads to quadratic PROMs that deliver the same accuracy as their traditional counterparts using however a much smaller dimension -- typically, $n_2 \sim \sqrt n_1$, where $n_2$ and $n_1$ denote the dimensions of the quadratic and traditional PROMs, respectively. The computational advantages of the proposed high-order approach to nonlinear PMOR over the traditional approach are highlighted for the detached-eddy simulation-based prediction of the Ahmed body turbulent wake flow, which is a popular CFD benchmark problem in the automotive industry. For a fixed accuracy level, these advantages include: a reduction of the total offline computational cost by a factor greater than five; a reduction of its online wall clock time by a factor greater than 32; and a reduction of the wall clock time of the underlying high-dimensional model by a factor greater than two orders of magnitude.
Configuration Path Control
Venue: arXiv cs.RO 2022
Year: 2022

Reinforcement learning methods often produce brittle policies -- policies that perform well during training, but generalize poorly beyond their direct training experience, thus becoming unstable under small disturbances. To address this issue, we propose a method for stabilizing a control policy in the space of configuration paths. It is applied post-training and relies purely on the data produced during training, as well as on an instantaneous control-matrix estimation. The approach is evaluated empirically on a planar bipedal walker subjected to a variety of perturbations. The control policies obtained via reinforcement learning are compared against their stabilized counterparts. Across different experiments, we find two- to four-fold increase in stability, when measured in terms of the perturbation amplitudes. We also provide a zero-dynamics interpretation of our approach.
Adversarial Robustness through the Lens of Convolutional Filters
Venue: arXiv cs.CV 2022
Year: 2022

Deep learning models are intrinsically sensitive to distribution shifts in the input data. In particular, small, barely perceivable perturbations to the input data can force models to make wrong predictions with high confidence. An common defense mechanism is regularization through adversarial training which injects worst-case perturbations back into training to strengthen the decision boundaries, and to reduce overfitting. In this context, we perform an investigation of 3x3 convolution filters that form in adversarially-trained models. Filters are extracted from 71 public models of the linf-RobustBench CIFAR-10/100 and ImageNet1k leaderboard and compared to filters extracted from models built on the same architectures but trained without robust regularization. We observe that adversarially-robust models appear to form more diverse, less sparse, and more orthogonal convolution filters than their normal counterparts. The largest differences between robust and normal models are found in the deepest layers, and the very first convolution layer, which consistently and predominantly forms filters that can partially eliminate perturbations, irrespective of the architecture. Data & Project website: https://github.com/paulgavrikov/cvpr22w_RobustnessThroughTheLens
Pareto-optimal clustering with the primal deterministic information bottleneck
Venue: arXiv cs.LG 2022
Year: 2022

At the heart of both lossy compression and clustering is a trade-off between the fidelity and size of the learned representation. Our goal is to map out and study the Pareto frontier that quantifies this trade-off. We focus on the optimization of the Deterministic Information Bottleneck (DIB) objective over the space of hard clusterings. To this end, we introduce the primal DIB problem, which we show results in a much richer frontier than its previously studied Lagrangian relaxation when optimized over discrete search spaces. We present an algorithm for mapping out the Pareto frontier of the primal DIB trade-off that is also applicable to other two-objective clustering problems. We study general properties of the Pareto frontier, and we give both analytic and numerical evidence for logarithmic sparsity of the frontier in general. We provide evidence that our algorithm has polynomial scaling despite the super-exponential search space, and additionally, we propose a modification to the algorithm that can be used where sampling noise is expected to be significant. Finally, we use our algorithm to map the DIB frontier of three different tasks: compressing the English alphabet, extracting informative color classes from natural images, and compressing a group theory-inspired dataset, revealing interesting features of frontier, and demonstrating how the structure of the frontier can be used for model selection with a focus on points previously hidden by the cloak of the convex hull.
On the Sustainability of Lightweight Cryptography Based on PUFs Implemented on NAND Flash Memories Using Programming Disturbances
Venue: arXiv cs.CR 2022
Year: 2022

In this work, we examine the potential of Physical Unclonable Functions (PUFs) that have been implemented on NAND Flash memories using programming disturbances to act as sustainable primitives for the purposes of lightweight cryptography. In particular, we investigate the ability of such PUFs to tolerate temperature and voltage variations, and examine the current shortcomings of existing NAND-Flash-memory PUFs that are based on programming disturbances as well as how these could potentially be addressed in order to provide more robust and more sustainable security solutions.
C3KG: A Chinese Commonsense Conversation Knowledge Graph
Venue: arXiv cs.CL 2022
Year: 2022

Existing commonsense knowledge bases often organize tuples in an isolated manner, which is deficient for commonsense conversational models to plan the next steps. To fill the gap, we curate a large-scale multi-turn human-written conversation corpus, and create the first Chinese commonsense conversation knowledge graph which incorporates both social commonsense knowledge and dialog flow information. To show the potential of our graph, we develop a graph-conversation matching approach, and benchmark two graph-grounded conversational tasks.