OpenRaiser/Intern-Atlas huggingface dataset:
NeuroView-RNN: It's About Time
Venue: arXiv cs.LG 2022
Year: 2022

Recurrent Neural Networks (RNNs) are important tools for processing sequential data such as time-series or video. Interpretability is defined as the ability to be understood by a person and is different from explainability, which is the ability to be explained in a mathematical formulation. A key interpretability issue with RNNs is that it is not clear how each hidden state per time step contributes to the decision-making process in a quantitative manner. We propose NeuroView-RNN as a family of new RNN architectures that explains how all the time steps are used for the decision-making process. Each member of the family is derived from a standard RNN architecture by concatenation of the hidden steps into a global linear classifier. The global linear classifier has all the hidden states as the input, so the weights of the classifier have a linear mapping to the hidden states. Hence, from the weights, NeuroView-RNN can quantify how important each time step is to a particular decision. As a bonus, NeuroView-RNN also offers higher accuracy in many cases compared to the RNNs and their variants. We showcase the benefits of NeuroView-RNN by evaluating on a multitude of diverse time-series datasets.
A modification of the conjugate direction method for motion estimation
Venue: arXiv cs.CV 2022
Year: 2022

A comparative study of different block matching alternatives for motion estimation is presented. The study is focused on computational burden and objective measures on the accuracy of prediction. Together with existing algorithms several new variations have been tested. An interesting modification of the conjugate direction method previously related in literature is reported. This new algorithm shows a good trade-off between computational complexity and accuracy of motion vector estimation. Computational complexity is evaluated using a sequence of artificial images designed to incorporate a great variety of motion vectors. The performance of block matching methods has been measured in terms of the entropy in the error signal between the motion compensated and the original frames.
Sky Computing: Accelerating Geo-distributed Computing in Federated Learning
Venue: arXiv cs.LG 2022
Year: 2022

Federated learning is proposed by Google to safeguard data privacy through training models locally on users' devices. However, with deep learning models growing in size to achieve better results, it becomes increasingly difficult to accommodate the whole model on one single device. Thus, model parallelism is then used to divide the model weights among several devices. With this logic, the approach currently used evenly allocates weights among devices. However, in reality, a computation bottleneck may occur resulting from variant computing power of different users' devices. To address this problem, load balancing is needed to allocate the model weights based on the computational capability of the device. In this paper, we proposed Sky Computing, a load-balanced model parallelism framework to adaptively allocate the weights to devices. Sky Computing outperforms the baseline method by 55% in training time when training 160-layer BERT with 64 nodes. The source code can be found at https://github.com/hpcaitech/SkyComputing.
Attainability and Optimality: The Equalized Odds Fairness Revisited
Venue: arXiv cs.LG 2022
Year: 2022

Fairness of machine learning algorithms has been of increasing interest. In order to suppress or eliminate discrimination in prediction, various notions as well as approaches have been proposed to impose fairness. Given a notion of fairness, an essential problem is then whether or not it can always be attained, even if with an unlimited amount of data. This issue is, however, not well addressed yet. In this paper, focusing on the Equalized Odds notion of fairness, we consider the attainability of this criterion and, furthermore, if it is attainable, the optimality of the prediction performance under various settings. In particular, for prediction performed by a deterministic function of input features, we give conditions under which Equalized Odds can hold true; if the stochastic prediction is acceptable, we show that under mild assumptions, fair predictors can always be derived. For classification, we further prove that compared to enforcing fairness by post-processing, one can always benefit from exploiting all available features during training and get potentially better prediction performance while remaining fair. Moreover, while stochastic prediction can attain Equalized Odds with theoretical guarantees, we also discuss its limitation and potential negative social impacts.
Joint Program Partitioning and Resource Allocation for Completion Time Minimization in Multi-MEC Systems
Venue: arXiv cs.NI 2022
Year: 2022

This paper considers a practical mobile edge computing (MEC) system, where edge server does not pre-install the program required to perform user offloaded computing tasks. A partial program offloading (PPO) scheme is proposed, which can divide a user program into two parts, where the first part is executed by the user itself and the second part is transferred to an edge server for remote execution. However, the execution of the latter part requires the results of the previous part (called intermediate result) as the input. We aim to minimize the overall time consumption of a multi-server MEC system to complete all user offloaded tasks. It is modeled as a mixed integer nonlinear programming (MINLP) problem which considers user-and-server association, program partitioning, and communication resource allocation in a joint manner. An effective algorithm is developed to solve the problem by exploiting its structural features. First, the task completion time of a single server is minimized given the computing workload and available resource. Then, the working time of the edge servers are balanced by updating user-and-server association and communication resource allocation. Numerical results show that significant performance improvement can be achieved by the proposed scheme.
HMD-EgoPose: Head-Mounted Display-Based Egocentric Marker-Less Tool and Hand Pose Estimation for Augmented Surgical Guidance
Venue: arXiv cs.CV 2022
Year: 2022

The success or failure of modern computer-assisted surgery procedures hinges on the precise six-degree-of-freedom (6DoF) position and orientation (pose) estimation of tracked instruments and tissue. In this paper, we present HMD-EgoPose, a single-shot learning-based approach to hand and object pose estimation and demonstrate state-of-the-art performance on a benchmark dataset for monocular red-green-blue (RGB) 6DoF marker-less hand and surgical instrument pose tracking. Further, we reveal the capacity of our HMD-EgoPose framework for performant 6DoF pose estimation on a commercially available optical see-through head-mounted display (OST-HMD) through a low-latency streaming approach. Our framework utilized an efficient convolutional neural network (CNN) backbone for multi-scale feature extraction and a set of subnetworks to jointly learn the 6DoF pose representation of the rigid surgical drill instrument and the grasping orientation of the hand of a user. To make our approach accessible to a commercially available OST-HMD, the Microsoft HoloLens 2, we created a pipeline for low-latency video and data communication with a high-performance computing workstation capable of optimized network inference. HMD-EgoPose outperformed current state-of-the-art approaches on a benchmark dataset for surgical tool pose estimation, achieving an average tool 3D vertex error of 11.0 mm on real data and furthering the progress towards a clinically viable marker-free tracking strategy. Through our low-latency streaming approach, we achieved a round trip latency of 199.1 ms for pose estimation and augmented visualization of the tracked model when integrated with the OST-HMD. Our single-shot learned approach was robust to occlusion and complex surfaces and improved on current state-of-the-art approaches to marker-less tool and hand pose estimation.
Interpolation-based Contrastive Learning for Few-Label Semi-Supervised Learning
Venue: arXiv cs.CV 2022
Year: 2022

Semi-supervised learning (SSL) has long been proved to be an effective technique to construct powerful models with limited labels. In the existing literature, consistency regularization-based methods, which force the perturbed samples to have similar predictions with the original ones have attracted much attention for their promising accuracy. However, we observe that, the performance of such methods decreases drastically when the labels get extremely limited, e.g., 2 or 3 labels for each category. Our empirical study finds that the main problem lies with the drifting of semantic information in the procedure of data augmentation. The problem can be alleviated when enough supervision is provided. However, when little guidance is available, the incorrect regularization would mislead the network and undermine the performance of the algorithm. To tackle the problem, we (1) propose an interpolation-based method to construct more reliable positive sample pairs; (2) design a novel contrastive loss to guide the embedding of the learned network to change linearly between samples so as to improve the discriminative capability of the network by enlarging the margin decision boundaries. Since no destructive regularization is introduced, the performance of our proposed algorithm is largely improved. Specifically, the proposed algorithm outperforms the second best algorithm (Comatch) with 5.3% by achieving 88.73% classification accuracy when only two labels are available for each class on the CIFAR-10 dataset. Moreover, we further prove the generality of the proposed method by improving the performance of the existing state-of-the-art algorithms considerably with our proposed strategy.
Support Recovery in Mixture Models with Sparse Parameters
Venue: arXiv cs.LG 2022
Year: 2022

Mixture models are widely used to fit complex and multimodal datasets. In this paper we study mixtures with high dimensional sparse latent parameter vectors and consider the problem of support recovery of those vectors. While parameter learning in mixture models is well-studied, the sparsity constraint remains relatively unexplored. Sparsity of parameter vectors is a natural constraint in variety of settings, and support recovery is a major step towards parameter estimation. We provide efficient algorithms for support recovery that have a logarithmic sample complexity dependence on the dimensionality of the latent space. Our algorithms are quite general, namely they are applicable to 1) mixtures of many different canonical distributions including Uniform, Poisson, Laplace, Gaussians, etc. 2) Mixtures of linear regressions and linear classifiers with Gaussian covariates under different assumptions on the unknown parameters. In most of these settings, our results are the first guarantees on the problem while in the rest, our results provide improvements on existing works.
A Holistic View on Data Protection for Sharing, Communicating, and Computing Environments: Taxonomy and Future Directions
Venue: arXiv cs.CR 2022
Year: 2022

The data is an important asset of an organization and it is essential to keep this asset secure. It requires security in whatever state is it i.e. data at rest, data in use, and data in transit. There is a need to pay more attention to it when the third party is included i.e. when the data is stored in the cloud then it requires more security. Since confidential data can reside on a variety of computing devices (physical servers, virtual servers, databases, file servers, PCs, point-of-sale devices, flash drives, and mobile devices) and move through a variety of network access points (wireline, wireless, VPNs, etc.), there is a need of solutions or mechanism that can tackle the problem of data loss, data recovery and data leaks. In this context, the paper presents a holistic view of data protection for sharing and communicating environments for any type of organization. A taxonomy of data leakage protection systems and major challenges faced while protecting confidential data are discussed. Data protection solutions, Data Leakage Protection System's analysis techniques, and, a thorough analysis of existing state-of-the-art contributions empowering machine learning-based approaches are entailed. Finally, the paper explores and concludes various critical emerging challenges and future research directions concerning data protection.
Fine-grained TLS services classification with reject option
Venue: arXiv cs.LG 2022
Year: 2022

The recent success and proliferation of machine learning and deep learning have provided powerful tools, which are also utilized for encrypted traffic analysis, classification, and threat detection in computer networks. These methods, neural networks in particular, are often complex and require a huge corpus of training data. Therefore, this paper focuses on collecting a large up-to-date dataset with almost 200 fine-grained service labels and 140 million network flows extended with packet-level metadata. The number of flows is three orders of magnitude higher than in other existing public labeled datasets of encrypted traffic. The number of service labels, which is important to make the problem hard and realistic, is four times higher than in the public dataset with the most class labels. The published dataset is intended as a benchmark for identifying services in encrypted traffic. Service identification can be further extended with the task of "rejecting" unknown services, i.e., the traffic not seen during the training phase. Neural networks offer superior performance for tackling this more challenging problem. To showcase the dataset's usefulness, we implemented a neural network with a multi-modal architecture, which is the state-of-the-art approach, and achieved 97.04% classification accuracy and detected 91.94% of unknown services with 5% false positive rate.
Proactive Libraries: Enforcing Correct Behaviors in Android Apps
Venue: arXiv cs.SE 2022
Year: 2022

The Android framework provides a rich set of APIs that can be exploited by developers to build their apps. However, the rapid evolution of these APIs jointly with the specific characteristics of the lifecycle of the Android components challenge developers, who may release apps that use APIs incorrectly. In this demo, we present Proactive Libraries, a tool that can be used to decorate regular libraries with the capability of proactively detecting and healing API misuses at runtime. Proactive Libraries blend libraries with multiple proactive modules that collect data, check the compliance of API usages with correctness policies, and heal executions as soon as the possible violation of a policy is detected. The results of our evaluation with 27 possible API misuses show the effectiveness of Proactive Libraries in correcting API misuses with negligible runtime overhead.
TriggerCit: Early Flood Alerting using Twitter and Geolocation -- a comparison with alternative sources
Venue: arXiv cs.CY 2022
Year: 2022

Rapid impact assessment in the immediate aftermath of a natural disaster is essential to provide adequate information to international organisations, local authorities, and first responders. Social media can support emergency response with evidence-based content posted by citizens and organisations during ongoing events. In the paper, we propose TriggerCit: an early flood alerting tool with a multilanguage approach focused on timeliness and geolocation. The paper focuses on assessing the reliability of the approach as a triggering system, comparing it with alternative sources for alerts, and evaluating the quality and amount of complementary information gathered. Geolocated visual evidence extracted from Twitter by TriggerCit was analysed in two case studies on floods in Thailand and Nepal in 2021.
Scenario Parameter Generation Method and Scenario Representativeness Metric for Scenario-Based Assessment of Automated Vehicles
Venue: arXiv cs.RO 2022
Year: 2022

The development of assessment methods for the performance of Automated Vehicles (AVs) is essential to enable the deployment of automated driving technologies, due to the complex operational domain of AVs. One candidate is scenario-based assessment, in which test cases are derived from real-world road traffic scenarios obtained from driving data. Because of the high variety of the possible scenarios, using only observed scenarios for the assessment is not sufficient. Therefore, methods for generating additional scenarios are necessary. Our contribution is twofold. First, we propose a method to determine the parameters that describe the scenarios to a sufficient degree without relying on strong assumptions on the parameters that characterize the scenarios. By estimating the probability density function (pdf) of these parameters, realistic parameter values can be generated. Second, we present the Scenario Representativeness (SR) metric based on the Wasserstein distance, which quantifies to what extent the scenarios with the generated parameter values are representative of real-world scenarios while covering the actual variety found in the real-world scenarios. A comparison of our proposed method with methods relying on assumptions of the scenario parametrization and pdf estimation shows that the proposed method can automatically determine the optimal scenario parametrization and pdf estimation. Furthermore, we demonstrate that our SR metric can be used to choose the (number of) parameters that best describe a scenario. The presented method is promising, because the parameterization and pdf estimation can directly be applied to already available importance sampling strategies for accelerating the evaluation of AVs.
Construction of a bi-infinite power free word with a given factor and a non-recurrent letter
Venue: arXiv cs.FL 2022
Year: 2022

Let $L_{k,α}^{\mathbb{Z}}$ denote the set of all bi-infinite $α$-power free words over an alphabet with $k$ letters, where $α$ is a positive rational number and $k$ is positive integer. We prove that if $α\geq 5$, $k\geq 3$, $v\in L_{k,α}^{\mathbb{Z}}$, and $w$ is a finite factor of $v$, then there are $\widetilde v\in L_{k,α}^{\mathbb{Z}}$ and a letter $x$ such that $w$ is a factor of $\widetilde v$ and $x$ has only a finitely many occurrences in $\widetilde v$.
Metacognitive Agents for Ethical Decision Support: Conceptual Model and Research Roadmap
Venue: arXiv cs.AI 2022
Year: 2022

An ethical value-action gap exists when there is a discrepancy between intentions and actions. This discrepancy may be caused by social and structural obstacles as well as cognitive biases. Computational models of cognition and affect can provide insights into the value-action gap and how it can be reduced. In particular, metacognition ("thinking about thinking") plays an important role in many of these models as a mechanism for self-regulation and reasoning about mental attitudes. This paper outlines a roadmap for translating cognitive-affective models into assistant agents to help make value-aligned decisions.
AFFDEX 2.0: A Real-Time Facial Expression Analysis Toolkit
Venue: arXiv cs.CV 2022
Year: 2022

In this paper we introduce AFFDEX 2.0 - a toolkit for analyzing facial expressions in the wild, that is, it is intended for users aiming to; a) estimate the 3D head pose, b) detect facial Action Units (AUs), c) recognize basic emotions and 2 new emotional states (sentimentality and confusion), and d) detect high-level expressive metrics like blink and attention. AFFDEX 2.0 models are mainly based on Deep Learning, and are trained using a large-scale naturalistic dataset consisting of thousands of participants from different demographic groups. AFFDEX 2.0 is an enhanced version of our previous toolkit [1], that is capable of tracking efficiently faces at more challenging conditions, detecting more accurately facial expressions, and recognizing new emotional states (sentimentality and confusion). AFFDEX 2.0 can process multiple faces in real time, and is working across the Windows and Linux platforms.
Light Robust Monocular Depth Estimation For Outdoor Environment Via Monochrome And Color Camera Fusion
Venue: arXiv cs.CV 2022
Year: 2022

Depth estimation plays a important role in SLAM, odometry, and autonomous driving. Especially, monocular depth estimation is profitable technology because of its low cost, memory, and computation. However, it is not a sufficiently predicting depth map due to a camera often failing to get a clean image because of light conditions. To solve this problem, various sensor fusion method has been proposed. Even though it is a powerful method, sensor fusion requires expensive sensors, additional memory, and high computational performance. In this paper, we present color image and monochrome image pixel-level fusion and stereo matching with partially enhanced correlation coefficient maximization. Our methods not only outperform the state-of-the-art works across all metrics but also efficient in terms of cost, memory, and computation. We also validate the effectiveness of our design with an ablation study.
"splink" is happy and "phrouth" is scary: Emotion Intensity Analysis for Nonsense Words
Venue: arXiv cs.CL 2022
Year: 2022

People associate affective meanings to words - "death" is scary and sad while "party" is connotated with surprise and joy. This raises the question if the association is purely a product of the learned affective imports inherent to semantic meanings, or is also an effect of other features of words, e.g., morphological and phonological patterns. We approach this question with an annotation-based analysis leveraging nonsense words. Specifically, we conduct a best-worst scaling crowdsourcing study in which participants assign intensity scores for joy, sadness, anger, disgust, fear, and surprise to 272 non-sense words and, for comparison of the results to previous work, to 68 real words. Based on this resource, we develop character-level and phonology-based intensity regressors. We evaluate them on both nonsense words and real words (making use of the NRC emotion intensity lexicon of 7493 words), across six emotion categories. The analysis of our data reveals that some phonetic patterns show clear differences between emotion intensities. For instance, s as a first phoneme contributes to joy, sh to surprise, p as last phoneme more to disgust than to anger and fear. In the modelling experiments, a regressor trained on real words from the NRC emotion intensity lexicon shows a higher performance (r = 0.17) than regressors that aim at learning the emotion connotation purely from nonsense words. We conclude that humans do associate affective meaning to words based on surface patterns, but also based on similarities to existing words ("juy" to "joy", or "flike" to "like").
Measuring CLEVRness: Blackbox testing of Visual Reasoning Models
Venue: arXiv cs.LG 2022
Year: 2022

How can we measure the reasoning capabilities of intelligence systems? Visual question answering provides a convenient framework for testing the model's abilities by interrogating the model through questions about the scene. However, despite scores of various visual QA datasets and architectures, which sometimes yield even a super-human performance, the question of whether those architectures can actually reason remains open to debate. To answer this, we extend the visual question answering framework and propose the following behavioral test in the form of a two-player game. We consider black-box neural models of CLEVR. These models are trained on a diagnostic dataset benchmarking reasoning. Next, we train an adversarial player that re-configures the scene to fool the CLEVR model. We show that CLEVR models, which otherwise could perform at a human level, can easily be fooled by our agent. Our results put in doubt whether data-driven approaches can do reasoning without exploiting the numerous biases that are often present in those datasets. Finally, we also propose a controlled experiment measuring the efficiency of such models to learn and perform reasoning.