Reviewerly/exHarmony huggingface dataset:
Real-Time Task Scheduling for Machine Perception in In Intelligent Cyber-Physical Systems

This paper explores <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">criticality-based real-time scheduling</i> of neural-network-based machine inference pipelines in cyber-physical systems (CPS) to mitigate the effect of algorithmic priority inversion. We specifically focus on the perception subsystem, an important subsystem feeding other components (e.g., planning and control). In general, priority inversion occurs in real-time systems when computations that are of lower priority are performed together with or ahead of those that are of higher priority. In current machine perception software, significant priority inversion occurs because <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">resource allocation</i> to the underlying neural network models does not differentiate between critical and less critical data within a scene. To remedy this problem, in recent work, we proposed an architecture to partition the input data into regions of different criticality, then formulated a utility-based optimization problem to batch and schedule their processing in a manner that maximizes confidence in perception results, subject to criticality-based time constraints. This journal extension matures the work in several directions: (i) We extend confidence maximization to a generalized utility optimization formulation that accounts for criticality in the utility function itself, offering finer-grained control over resource allocation within the perception pipeline; (ii) we further instantiate and compare two different criticality metrics (distance-based and relative velocity-based) to understand their relative advantages; and (iii) we explore the limitations of the approach, specifically how inaccuracies in criticality-based attention cueing affect performance. All experiments are conducted on the NVIDIA Jetson AGX Xavier platform with a real-world driving dataset.
Directions for 3D User Interface Research from Consumer VR Games

With the continuing development of affordable immersive virtual reality (VR) systems, there is now a growing market for consumer content. The current form of consumer systems is not dissimilar to the lab-based VR systems of the past 30 years: the primary input mechanism is a head-tracked display and one or two tracked hands with buttons and joysticks on hand-held controllers. Over those 30 years, a very diverse academic literature has emerged that covers design and ergonomics of 3D user interfaces (3DUIs). However, the growing consumer market has engaged a very broad range of creatives that have built a very diverse set of designs. Sometimes these designs adopt findings from the academic literature, but other times they experiment with completely novel or counter-intuitive mechanisms. In this paper and its online adjunct, we report on novel 3DUI design patterns that are interesting from both design and research perspectives: they are highly novel, potentially broadly re-usable and/or suggest interesting avenues for evaluation. The supplemental material, which is a living document, is a crowd-sourced repository of interesting patterns. This paper is a curated snapshot of those patterns that were considered to be the most fruitful for further elaboration.
Case for 5G-aware video streaming applications

Recent measurement studies show that commercial mmWave 5G can indeed offer ultra-high bandwidth (up to 2 Gbps), capable of supporting bandwidth-intensive applications such as ultra-HD (UHD) 4K/8K and volumetric video streaming on mobile devices. However, mmWave 5G also exhibits highly variable throughput performance and incurs frequent handoffs (e.g., between 5G and 4G), due to its directional nature, signal blockage and other environmental factors, especially when the device is mobile. All these issues make it difficult for applications to achieve high Quality of Experience (QoE). In this paper, we advance several new mechanisms to tackle the challenges facing UHD video streaming applications over 5G networks, thereby making them {\em 5G-aware}. We argue for the need to employ machine learning (ML) for effective throughput prediction to aid applications in intelligent bitrate adaptation. Furthermore, we advocate {\em adaptive content bursting}, and {\em dynamic radio (band) switching} to allow the 5G radio network to fully utilize the available radio resources under good channel/beam conditions, whereas dynamically switched radio channels/bands (e.g., from 5G high-band to low-band, or 5G to 4G) to maintain session connectivity and ensure a minimal bitrate. We conduct initial evaluation using real-world 5G throughput measurement traces. Our results show these mechanisms can help minimize, if not completely eliminate, video stalls, despite wildly varying 5G throughput.
Universal multi-Source domain adaptation for image classification

Unsupervised domain adaptation (DA) enables intelligent models to learn transferable knowledge from a labeled source domain and adapt to a similar but unlabeled target domain. Studies showed that knowledge could be transferred from one source domain to another unknown target domain, called Universal DA (UDA). However, there is often more than one source domain in the real-world application to be exploited for DA. In this paper, we formally propose a more general domain adaptation setting for image classification, universal multi-source DA (UMDA), where the label sets of multiple source domains can be different, and the label set of the target domain is completely unknown. The main challenge in UMDA is to identify the common label set among each source and target domain and keep the model scalable as the number of source domains increases. In the face of this challenge, we propose a universal multi-source adaptation network (UMAN) to solve the DA problem without increasing the complexity of the model in various UMDA settings. In UMAN, the reliability of each known class belonging to the common label set is estimated via a novel pseudo-margin vector and its weighted form, which helps adversarial training better align the distributions of multiple source domains and target domain. Moreover, the theoretical guarantee for UMAN is also provided. Massive experimental results show that existing UDA and multi-source DA (MDA) methods cannot be directly deployed to UMDA, and the proposed UMAN achieves the state-of-the-art performance in various UMDA settings.
Logical characterizations of fuzzy bisimulations in fuzzy modal logics over residuated lattices

There are two kinds of bisimulation, namely crisp and fuzzy, between fuzzy structures such as fuzzy automata, fuzzy labeled transition systems, fuzzy Kripke models and fuzzy interpretations in description logics. Fuzzy bisimulations between fuzzy automata over a complete residuated lattice have been introduced by Ćirić et al. in 2012. Logical characterizations of fuzzy bisimulations between fuzzy Kripke models (respectively, fuzzy interpretations in description logics) over the residuated lattice [0,1] with the Gödel t-norm have been provided by Fan in 2015 (respectively, Nguyen et al. in 2020). There was the lack of logical characterizations of fuzzy bisimulations between fuzzy graph-based structures over a general residuated lattice, as well as over the residuated lattice [0,1] with the Łukasiewicz or product t-norm. In this article, we provide and prove logical characterizations of fuzzy bisimulations in fuzzy modal logics over residuated lattices. The considered logics are the fuzzy propositional dynamic logic and its fragments. Our logical characterizations concern invariance of formulas under fuzzy bisimulations and the Hennessy-Milner property of fuzzy bisimulations. They can be reformulated for other fuzzy structures such as fuzzy labeled transition systems and fuzzy interpretations in description logics.
Publicly verifiable outsourced data migration scheme supporting efficient integrity checking

With the rapid development of cloud storage, an increasing number of data owners prefer to store their large-scale data on the remote cloud for greatly reducing the burden of maintaining the data by themselves. Since different cloud storage service providers offer distinct quality of data storage services, secure outsourced data migration becomes a fundamental requirement for data owners to change their cloud storage service providers. As a result, how to guarantee the correctness and integrity of the outsourced data blocks when they are being migrated from one semi-trusted cloud host to another becomes a primary concern of data owners. To solve this problem, we propose a blockchain-based publicly verifiable data migration scheme supporting efficient data integrity checking for cloud storage in this paper. Specifically, the data owner can dynamically change the cloud storage service providers and securely migrate the outsourced data from one cloud host to another, without retrieving the transferred data from the original cloud host, and checking the transferred data integrity in the target cloud host. By making use of blockchain, our proposed scheme can simultaneously achieve data migration and integrity checking without interacting with a third party auditor, thus avoiding the problems of service interruption and privacy leakage caused by the single-point-of-failure of the third party auditor, which is much better than the previous schemes. Meanwhile, our proposed scheme is provable secure in the random oracle model. Finally, we also develop a prototype implementation of our proposed scheme and compare the efficiency with some existing schemes. The experimental result demonstrates that our proposed scheme is more efficient and practical.
Bilinear Graph Networks for Visual Question Answering

This article revisits the bilinear attention networks (BANs) in the visual question answering task from a graph perspective. The classical BANs build a bilinear attention map to extract the joint representation of words in the question and objects in the image but lack fully exploring the relationship between words for complex reasoning. In contrast, we develop bilinear graph networks to model the context of the joint embeddings of words and objects. Two kinds of graphs are investigated, namely, image-graph and question-graph. The image-graph transfers features of the detected objects to their related query words, enabling the output nodes to have both semantic and factual information. The question-graph exchanges information between these output nodes from image-graph to amplify the implicit yet important relationship between objects. These two kinds of graphs cooperate with each other, and thus, our resulting model can build the relationship and dependency between objects, which leads to the realization of multistep reasoning. Experimental results on the VQA v2.0 validation dataset demonstrate the ability of our method to handle complex questions. On the test-std set, our best single model achieves state-of-the-art performance, boosting the overall accuracy to 72.56%, and we are one of the top-two entries in the VQA Challenge 2020.
Forward and inverse reinforcement learning sharing network weights and hyperparameters

This paper proposes model-free imitation learning named Entropy-Regularized Imitation Learning (ERIL) that minimizes the reverse Kullback-Leibler (KL) divergence. ERIL combines forward and inverse reinforcement learning (RL) under the framework of an entropy-regularized Markov decision process. An inverse RL step computes the log-ratio between two distributions by evaluating two binary discriminators. The first discriminator distinguishes the state generated by the forward RL step from the expert's state. The second discriminator, which is structured by the theory of entropy regularization, distinguishes the state-action-next-state tuples generated by the learner from the expert ones. One notable feature is that the second discriminator shares hyperparameters with the forward RL, which can be used to control the discriminator's ability. A forward RL step minimizes the reverse KL estimated by the inverse RL step. We show that minimizing the reverse KL divergence is equivalent to finding an optimal policy. Our experimental results on MuJoCo-simulated environments and vision-based reaching tasks with a robotic arm show that ERIL is more sample-efficient than the baseline methods. We apply the method to human behaviors that perform a pole-balancing task and describe how the estimated reward functions show how every subject achieves her goal.
Multilevel weather detection based on images: a machine learning approach with histogram of oriented gradient and local binary pattern-based features

The primary objective of this study was to develop a trajectory-level weather detection system capable of providing real-time weather information at the road surface level using only a single video camera. Two texture-based features, including histogram of oriented gradient (HOG) and local binary pattern (LBP), were extracted from images and used as classification parameters to train the weather detection models using several machine learning classifiers, such as gradient boosting (GB), random forest (RF), and support vector machine (SVM). In addition, a unique multilevel model, based on a hierarchical structure, was also proposed to increase detection accuracy. Evaluation results revealed that the multilevel model provided an overall accuracy of 89.2%, which is 3.2%, 7.5%, and 7.9% higher compared to the SVM, RF, and GB model, respectively, using the HOG features. Considering the LBP features, the multilevel model also produced the best performance with an overall accuracy of 91%, which is 1.6%, 8.6%, and 9% higher compared to the SVM, RF, and GB models, respectively. A sensitivity analysis using the proposed multilevel model revealed that the classification accuracy improved with the increasing number of HOG and LBP features at the expense of more computational powers. The proposed weather detection method is cost-efficient and can be made widely available mainly due to the recent booming of smartphone cameras and can be used to expand and update the current weather-based variable speed limit (VSL) systems in a connected vehicle (CV) environment.
Learning to survive amidst nested crises: can the coronavirus pandemic help us change educational practices to prepare for the impending eco-crisis?

The ongoing ecological crisis and the more recent Coronavirus crisis challenge the grand narrative of Enlightenment that human beings are 'masters of nature'. For millennia, human social learning has allowed Homo sapiens to outpace most of our competitor creatures and live a comfortable life, but this competitive success has resulted in cataclysmic failure for the ecosystem. However, people's unique ability to learn gives us hope that we can overcome the nested crises, or learn to live with them. What is required is not more knowledge, but instead, collective learning to change practices, institutionalized in educational processes. Drawing on the theory of practice architectures, this paper discusses how education can help to form a new generation of children, young people, and adults equipped for the new post-Corona world, and equipped to respond appropriately to the eco-crisis. This requires significant changes to existing arrangements of education systems. What is needed is new practice architectures – new conditions of possibility – under which human beings can learn to live sustainably within the community of life on Earth.
LINKTELLER: Recovering Private Edges from Graph Neural Networks via Influence Analysis

Graph structured data have enabled several successful applications such as recommendation systems and traffic prediction, given the rich node features and edges information. However, these high-dimensional features and high-order adjacency information are usually heterogeneous and held by different data holders in practice. Given such vertical data partition e.g., one data holder will only own either the node features or edge information), different data holders have to develop efficient joint training protocols rather than directly transferring data to each other due to privacy concerns. In this paper, we focus on the edge privacy, and consider a training scenario where the data holder Bob with node features will first send training node features to Alice who owns the adjacency information. Alice will then train a graph neural network (GNN) with the joint information and provide an inference API to Bob. During inference time, Bob is able to provide test node features and query the API to obtain the predictions for test nodes. Under this setting, we first propose a privacy attack LINKTELLER via influence analysis to infer the private edge information held by Alice via designing adversarial queries for Bob. We then empirically show that LINKTELLER is able to recover a significant amount of private edges in different settings, both including inductive (8 datasets) and transductive (3 datasets), under different graph densities, significantly outperforming existing baselines. To further evaluate the privacy leakage for edges, we adapt an existing algorithm for differentially private graph convolutional network (DP GCN) training as well as propose a new DP GCN mechanism LAPGRAPH based on Laplacian mechanism to evaluate LINKTELLER. We show that these DP GCN mechanisms are not always resilient against LINKTELLER empirically under mild privacy guarantees $(\varepsilon\gt 5)$. Our studies will shed light on future research towards designing more resilient privacy-preserving GCN models; in the meantime, provide an in-depth understanding about the tradeoff between GCN model utility and robustness against potential privacy attacks.
Repeat victimization by website defacement: An empirical test of premises from an environmental criminology perspective

Repeat victimization has been widely studied from the perspective of environmental criminology for several decades. During this period, criminologists have identified a set of repeat victimization premises that are observed for many crimes; however, it is unknown whether these premises are also valid for cybercrime. In this study we rely on more than 9 million Zone-H data records from 2010 to 2017 to test whether these premises apply for the cybercrime of website defacement. We show that the phenomenon of repeat victimization is also observed in defaced cyber places (i.e. websites). In particular, we found that repeats contributed little to crime rates, that repeats occurred even several years after the original incident, that they were committed disproportionately by prolific offenders, and that few offenders returned to victimize previous targets. The results suggest that some traditional premises of repeat victimization may also be valid for understanding cybercrime events such as website defacement, implying that environmental criminology theories also constitute a useful framework for cybercrime analysis. The implications of these results in terms of criminological theory, cybercrime prevention, and the limitations derived from the use of Zone-H data are discussed.
Auto-Parsing Network for Image Captioning and Visual Question Answering

We propose an Auto-Parsing Network (APN) to discover and exploit the input data’s hidden tree structures for improving the effectiveness of the Transformer-based vision-language systems. Specifically, we impose a Probabilistic Graphical Model (PGM) parameterized by the attention operations on each self-attention layer to incorporate sparse assumption. We use this PGM to softly segment an input sequence into a few clusters where each cluster can be treated as the parent of the inside entities. By stacking these PGM constrained self-attention layers, the clusters in a lower layer compose into a new sequence, and the PGM in a higher layer will further segment this sequence. Iteratively, a sparse tree can be implicitly parsed, and this tree’s hierarchical knowledge is incorporated into the transformed embeddings, which can be used for solving the target vision-language tasks. Specifically, we showcase that our APN can strengthen Transformer based networks in two major vision-language tasks: Captioning and Visual Question Answering. Also, a PGM probability-based parsing algorithm is developed by which we can discover what the hidden structure of input is during the inference.
WSUIE: Weakly Supervised Underwater Image Enhancement for Improved Visual Perception

Underwater images inevitably suffer from degradation and blur due to the scattering and absorption of light as it propagates through the water, which hinders the development of underwater visual perception. Existing deep underwater image enhancement methods mainly rely on the strong supervision of a large-scale dataset composed of aligned raw/enhanced underwater image pairs for model training. However, aligned image pairs are not available in most underwater scenes. This work aims to address this problem by proposing a novel weakly supervised underwater image enhancement (named WSUIE) method. Firstly, a novel generative adversarial network (GAN)-based architecture is designed to enhance underwater images by unpaired image-to-image transformation from domain <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</b> (raw underwater images) to domain <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Y</b> (arbitrary high-quality images), which alleviates the need for aligned underwater image pairs. Then, a new objective function is formulated by exploring intrinsic depth information of underwater images to increase the depth sensitivity of our method. In addition, a dataset with unaligned image pairs (named UUIE) is provided for the model training. Many qualitative and quantitative evaluations of the WSUIE method are performed on this dataset, and the results show that this method can provide improved visual perception performance while enhancing visual quality of underwater images.
Comparing Swarm Intelligence Algorithms for Dimension Reduction in Machine Learning

Nowadays, the high-dimensionality of data causes a variety of problems in machine learning. It is necessary to reduce the feature number by selecting only the most relevant of them. Different approaches called Feature Selection are used for this task. In this paper, we propose a Feature Selection method that uses Swarm Intelligence techniques. Swarm Intelligence algorithms perform optimization by searching for optimal points in the search space. We show the usability of these techniques for solving Feature Selection and compare the performance of five major swarm algorithms: Particle Swarm Optimization, Artificial Bee Colony, Invasive Weed Optimization, Bat Algorithm, and Grey Wolf Optimizer. The accuracy of a decision tree classifier was used to evaluate the algorithms. It turned out that the dimension of the data can be reduced about two times without a loss in accuracy. Moreover, the accuracy increased when abandoning redundant features. Based on our experiments GWO turned out to be the best. It has the highest ranking on different datasets, and its average iteration number to find the best solution is 30.8. ABC obtained the lowest ranking on high-dimensional datasets.