Reviewerly/exHarmony huggingface dataset:
Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction

We propose a multiscale spatio-temporal graph neural network (MST-GNN) to predict the future 3D skeleton-based human poses in an action-category-agnostic manner. The core of MST-GNN is a multiscale spatio-temporal graph that explicitly models the relations in motions at various spatial and temporal scales. Different from many previous hierarchical structures, our multiscale spatio-temporal graph is built in a data-adaptive fashion, which captures nonphysical, yet motion-based relations. The key module of MST-GNN is a multiscale spatio-temporal graph computational unit (MST-GCU) based on the trainable graph structure. MST-GCU embeds underlying features at individual scales and then fuses features across scales to obtain a comprehensive representation. The overall architecture of MST-GNN follows an encoder-decoder framework, where the encoder consists of a sequence of MST-GCUs to learn the spatial and temporal features of motions, and the decoder uses a graph-based attention gate recurrent unit (GA-GRU) to generate future poses. Extensive experiments are conducted to show that the proposed MST-GNN outperforms state-of-the-art methods in both short and long-term motion prediction on the datasets of Human 3.6M, CMU Mocap and 3DPW, where MST-GNN outperforms previous works by 5.33% and 3.67% of mean angle errors in average for short-term and long-term prediction on Human 3.6M, and by 11.84% and 4.71% of mean angle errors for short-term and long-term prediction on CMU Mocap, and by 1.13% of mean angle errors on 3DPW in average, respectively. We further investigate the learned multiscale graphs for interpretability.
Machine learning based liver disease diagnosis: A systematic review

The computer-based approach is required for the non-invasive detection of chronic liver diseases that are asymptomatic, progressive, and potentially fatal in nature. In this study, we review the computer-aided diagnosis of hepatic lesions in view of diffuse- and focal liver disorders. This survey mainly focuses on three image acquisition modalities: ultrasonography, computed tomography, and magnetic resonance imaging. We present the insightful analysis with pros and cons for each preliminary step, particularly preprocessing, attribute analysis, and classification techniques to accomplish clinical diagnostic tasks. In preprocessing, we explore and compare commonly used denoising, deblurring and segmentation methods. Denoising is mainly performed with nonlinear models. In contrast, deep neural networks are frequently applied for deblurring and automatic segmentation of region-of-interest. In attribute analysis, the most common approach comprises texture properties. For classification, the support vector machine is mainly utilized across three image acquisition modalities. However, comparative analysis shows the best performance is obtained by deep learning-based convolutional neural networks. Considering biopsy samples or pathological factors such as overall stage, margin, and differentiation can be helpful for improving the prediction performance. In addition, technique breakthrough is expected soon with advances in machine learning models to address data limitation problems and improve the prediction performance.
A Unified Cognitive Learning Framework for Adapting to Dynamic Environments and Tasks

Many machine learning frameworks have been proposed and used in wireless communications for realizing diverse goals. However, their incapability of adapting to dynamic wireless environments and tasks and of self-learning limit their extensive applications and achievable performance. Inspired by the great flexibility and adaptation of primate behaviors due to the brain cognitive mechanism, a unified cognitive learning (CL) framework is proposed for dynamic wireless environments and tasks. The mathematical framework for our proposed CL is established. Using the public and authoritative dataset, we demonstrate that our proposed CL framework has three advantages, namely, the capability of adapting to dynamic environments and tasks, the self-learning capability and the capability of &#x201C;good money driving out bad money&#x201D; by taking modulation recognition as an example. The proposed CL framework can enrich the current learning frameworks and widen the applications.
DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network

Shadow removal from a single image is generally still an open problem. Most existing learning-based methods use supervised learning and require a large number of paired images (shadow and corresponding non-shadow images) for training. A recent unsupervised method, Mask-ShadowGAN [13], addresses this limitation. However, it requires a binary mask to represent shadow regions, making it inapplicable to soft shadows. To address the problem, in this paper, we propose an unsupervised domain-classifier guided shadow removal network, DC-ShadowNet. Specifically, we propose to integrate a shadow/shadow-free domain classifier into a generator and its discriminator, enabling them to focus on shadow regions. To train our network, we introduce novel losses based on physics-based shadow-free chromaticity, shadow-robust perceptual features, and boundary smoothness. Moreover, we show that our unsupervised network can be used for test-time training that further improves the results. Our experiments show that all these novel components allow our method to handle soft shadows, and also to perform better on hard shadows both quantitatively and qualitatively than the existing state-of-the-art shadow removal methods.
Design of thermal imaging-based health condition monitoring and early fault detection technique for porcelain insulators using Machine learning

The inspection of insulator faults is an important task to prevent catastrophic failures in the operation of an electric substation. Manual inspection of overhead power line insulators can be very dangerous owning to the presence of high voltage in power sub-stations. Hence, in this paper, we present an infrared thermal (IRT) camera based non-invasive computer vision system for automatic monitoring and visual inspection of overhead on line power insulators. In the proposed work, initially, an optimal threshold method is applied to segment the region of interest (ROI) in IRT images. Subsequently, various geometrical, morphological, intensity and statistical features are computed from the segmented ROI, which are eventually utilized as an input to Gaussian kernel support vector machine to classify the different type of faults in insulator images. Computer vision based automatic inspection of insulators can play an important role from environment as well as human safety point of view. Timely inspection of insulators can ensure the environmental safety through prevention of fire that may be caused due to the insulator failures leading to the sudden breakdown of high-power lines. The proposed system achieved the true positive rate (TPR), and false negative rate (FNR) of 97.3%, and 2.66%, respectively. Whereas, the system obtained the Positive Predictive Value (PPV) and False Discovery Rates (FDR) of about 97% and 3%, respectively with the accuracy of 0.97 on the receiver operating characteristics (ROC) curve. The performance of the proposed method is compared with other existing state of the art methods and found that our method outperformed over them. Hence, we recommend proposed system to detect the severity of faults in IRT images long before any catastrophic failures take place at power sub-stations.
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models

We extend the task of composed image retrieval, where an input query consists of an image and short textual description of how to modify the image. Existing methods have only been applied to non-complex images within narrow domains, such as fashion products, thereby limiting the scope of study on in-depth visual reasoning in rich image and language contexts. To address this issue, we collect the Compose Image Retrieval on Real-life images (CIRR) dataset, which consists of over 36,000 pairs of crowd-sourced, open-domain images with human-generated modifying text. To extend current methods to the open-domain, we propose CIRPLANT, a transformer based model that leverages rich pre-trained vision-and-language (V&L) knowledge for modifying visual features conditioned on natural language. Retrieval is then done by nearest neighbor lookup on the modified features. We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while matching state-of-the-art accuracy on the existing narrow datasets, such as fashion. Together with the release of CIRR, we believe this work will inspire further research on composed image retrieval. Our dataset, code and pre-trained models are available at https://cuberick-orion.github.io/CIRR/.
A Deep Reinforcement Learning-Based Resource Management Game in Vehicular Edge Computing

Vehicular Edge Computing (VEC) is a promising paradigm that leverages the vehicles to offload computation tasks to the nearby VEC server with the aim of supporting the low latency vehicular application scenarios. Incentivizing VEC servers to participate in computation offloading activities and make full use of computation resources is of great importance to the success of intelligent transportation services. In this paper, we formulate the competitive interactions between the VEC servers and vehicles as a two-stage Stackelberg game with the VEC servers as the leader players and the vehicles as the followers. After obtaining the full information of vehicles, the VEC server calculates the unit price of computation resource. Given the unit prices announced by VEC server, the vehicles determine the amount of computation resource to purchase from VEC server. In the scenario that vehicles do not want to share their computation demands, a deep reinforcement learning based resource management scheme is proposed to maximize the profits of vehicles and VEC server. The extensive experimental results have demonstrated the effectiveness of our proposed resource management scheme based on Stackelberg game and deep reinforcement learning.
Machine Learning techniques for solar irradiation nowcasting: Cloud type classification forecast through satellite data and imagery

One of the most important modern challenges in making the renewable energy sources more reliable is the development of new tools to better manage their non programmable nature and avoid economic losses, to ensure compliance with network constraints and to improve the management of congestion. The solar energy at ground level exhibits a continuous variation in time and space. This fluctuation has a deterministic component generated by the movements of rotation and revolution of the earth, and a random one generated by weather conditions. Solar energy variations at ground level have a great influence on the output power of a photovoltaic plant, which can fluctuate significantly in short intervals due to the random component. This work presents a new model to detect in real time the clouds which potentially obstruct the sunrays directed to a specific geographic target. Moreover, a novel procedure for the forecasting of the clearness sky index on the target in the fifteen minutes is proposed, levereging on Machine Learning techniques, exploiting satellite and weather data.
Multi-view spectral clustering via constrained nonnegative embedding

Multi-view clustering has attracted much attention recently. Among all clustering approaches, spectral ones have gained much popularity thanks to an elaborated and solid theoretical foundation. A major limitation of spectral clustering based methods is that these methods only provide a non-linear projection of the data, to which an additional step of clustering is required. This can degrade the quality of the final clustering due to various factors such as the initialization process or outliers. To overcome these challenges, this paper presents a constrained version of a recent method called Multiview Spectral Clustering via integrating Nonnegative Embedding and Spectral Embedding. Besides retaining the advantages of this method, our proposed model integrates two types of constraints: (i) a consistent smoothness of the nonnegative embedding over all views and (ii) an orthogonality constraint over the columns of the nonnegative embedding. Experimental results on several real datasets show the superiority of the proposed approach.
STAR: A Structure-aware Lightweight Transformer for Real-time Image Enhancement

Image and video enhancement such as color constancy, low light enhancement, and tone mapping on smartphones is challenging, because high-quality images should be achieved efficiently with a limited resource budget. Unlike prior works that either used very deep CNNs or large Trans-former models, we propose a structure-aware lightweight Transformer, termed STAR, for real-time image enhancement. STAR is formulated to capture long-range dependencies between image patches, which naturally and implicitly captures the structural relationships of different regions in an image. STAR is a general architecture that can be easily adapted to different image enhancement tasks. Extensive experiments show that STAR can effectively boost the quality and efficiency of many tasks such as illumination enhancement, auto white balance, and photo retouching, which are indispensable components for image processing on smartphones. For example, STAR reduces model complexity and improves image quality compared to the recent state-of-the-art [19] on the MIT-Adobe FiveK dataset [7] (i.e., 1.8dB PSNR improvements with 25% parameters and 13% float operations.)
Ready Teacher One: Virtual and Augmented Reality Online Professional Development for K-12 School Teachers

Augmented reality (AR) and virtual reality (VR) are two technologies with the potential to enhance learning quality by activating episodic student memory. In this manuscript we outline the pedagogical benefits of AR and VR as well as the rationale, design, development, and pilot evaluation results from an in-service teacher online professional development program (OPD) on AR and VR linked with research activities. More specifically, we examined the “Augmented and Virtual Reality in Teaching and Learning” OPD course aimed at K-12 educators in a European country combining pedagogically informed methods such as inquiry-based learning and digital storytelling. Findings from a mixed-method research using questionnaires and focus groups indicate that AR- and VR-based pedagogical scenarios can be integrated effectively into everyday school teaching practice. The study also outlines implications and limitations that policy makers, education leaders, and educators need to consider for efficient institution-wide deployment of AR and VR technologies.
FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation

The RGB-Thermal (RGB-T) information for semantic segmentation has been extensively explored in recent years. However, most existing RGB-T semantic segmentation usually compromises spatial resolution to achieve real-time inference speed, which leads to poor performance. To better extract detail spatial information, we propose a two-stage Feature-Enhanced Attention Network (FEANet) for the RGB-T semantic segmentation task. Specifically, we introduce a Feature-Enhanced Attention Module (FEAM) to excavate and enhance multi-level features from both the channel and spatial views. Benefited from the proposed FEAM module, our FEANet can preserve the spatial information and shift more attention to high-resolution features from the fused RGB-T images. Extensive experiments on the urban scene dataset demonstrate that our FEANet outperforms other state-of-the-art (SOTA) RGB-T methods in terms of objective metrics and subjective visual comparison (+2.6% in global mAcc and +0.8% in global mIoU). For the 480 × 640 RGB-T test images, our FEANet can run with a real-time speed on an NVIDIA GeForce RTX 2080 Ti card.
PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition

Nowadays, scene text recognition has attracted more and more attention due to its various applications. Most state-of-the-art methods adopt an encoder-decoder framework with attention mechanism, which generates text autoregressively from left to right. Despite the convincing performance, the speed is limited because of the one-by-one decoding strategy. As opposed to autoregressive models, non-autoregressive models predict the results in parallel with a much shorter inference time, but the accuracy falls behind the autoregressive counterpart considerably. In this paper, we propose a Parallel, Iterative and Mimicking Network (PIMNet) to balance accuracy and efficiency. Specifically, PIMNet adopts a parallel attention mechanism to predict the text faster and an iterative generation mechanism to make the predictions more accurate. In each iteration, the context information is fully explored. To improve learning of the hidden layer, we exploit the mimicking learning in the training phase, where an additional autoregressive decoder is adopted and the parallel decoder mimics the autoregressive decoder with fitting outputs of the hidden layer. With the shared backbone between the two decoders, the proposed PIMNet can be trained end-to-end without pre-training. During inference, the branch of the autoregressive decoder is removed for a faster speed. Extensive experiments on public benchmarks demonstrate the effectiveness and efficiency of PIMNet. Our code is available in the supplementary material.
Federated Learning Over Wireless Device-to-Device Networks: Algorithms and Convergence Analysis

The proliferation of Internet-of-Things (IoT) devices and cloud-computing applications over siloed data centers is motivating renewed interest in the collaborative training of a shared model by multiple individual clients via federated learning (FL). To improve the communication efficiency of FL implementations in wireless systems, recent works have proposed compression and dimension reduction mechanisms, along with digital and analog transmission schemes that account for channel noise, fading, and interference. The prior art has mainly focused on star topologies consisting of distributed clients and a central server. In contrast, this paper studies FL over wireless device-to-device (D2D) networks by providing theoretical insights into the performance of digital and analog implementations of decentralized stochastic gradient descent (DSGD). First, we introduce generic digital and analog wireless implementations of communication-efficient DSGD algorithms, leveraging random linear coding (RLC) for compression and over-the-air computation (AirComp) for simultaneous analog transmissions. Next, under the assumptions of convexity and connectivity, we provide convergence bounds for both implementations. The results demonstrate the dependence of the optimality gap on the connectivity and on the signal-to-noise ratio (SNR) levels in the network. The analysis is corroborated by experiments on an image-classification task.
ROBIN: a Graph-Theoretic Approach to Reject Outliers in Robust Estimation using Invariants

Many estimation problems in robotics, computer vision, and learning require estimating unknown quantities in the face of outliers. Outliers are typically the result of incorrect data association or feature matching, and it is not uncommon to have problems where more than 90% of the measurements used for estimation are outliers. While current approaches for robust estimation (e.g., RANSAC or graduated non-convexity) are able to deal with moderate amounts of outliers, they fail to produce accurate estimates in the presence of many outliers. This paper develops an approach to prune outliers. First, we develop a theory of invariance that allows us to quickly check if a subset of measurements are mutually compatible without explicitly solving the corresponding estimation problem. Second, we develop a graph-theoretic framework, where measurements are modeled as vertices and mutual compatibility is captured by edges in a graph. We generalize existing results showing that the inliers form a clique in this compatibility graph and typically belong to the maximum clique. We also show that in practice the maximum k-core of the compatibility graph provides an approximation of the maximum clique, while being much faster to compute in large problems. The combination of these two contributions leads to ROBIN, our approach to Reject Outliers Based on INvariants, which allows us to quickly prune outliers in generic estimation problems. We demonstrate ROBIN in four geometric perception problems and show it boosts robustness of existing solvers (making them robust to more than 95% outliers), while running in milliseconds in large problems.
DeepHateExplainer: Explainable Hate Speech Detection in Under-resourced Bengali Language

In this paper, we propose an explainable approach for hate speech detection from the under-resourced Bengali language, which we called DeepHateExplainer. In our approach, Bengali texts are first comprehensively preprocessed, before classifying them into political, personal, geopolitical, and religious hates using a neural ensemble method of transformer-based neural architectures (i.e., monolingual Bangla BERT-base, multilingual BERT-cased/uncased, and XLM-RoBERTa). Subsequently, important (most and least) terms are identified using sensitivity analysis and layer-wise relevance propagation (LRP), before providing human-interpretable explanations <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> To foster reproducible research, we make available the data, source codes, models, and notebooks: https://github.com/rezacsedu/DeepHateExplainer. Finally, we compute comprehensiveness and sufficiency scores to measure the quality of explanations w.r.t faithfulness. Evaluations against machine learning (linear and tree-based models) and neural networks (i.e., CNN, Bi-LSTM, and Conv-LSTM with word embeddings) baselines yield F1-scores of 78%, 91%, 89%, and 84%, for political, personal, geopolitical, and religious hates, respectively, outperforming both ML and DNN baselines <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Read an extended version of this paper: https://arxiv.org/abs/2012.14353.