Reviewerly/exHarmony huggingface dataset:
Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

While recent studies on semi-supervised learning have shown remarkable progress in leveraging both labeled and unlabeled data, most of them presume a basic setting of the model is randomly initialized. In this work, we consider semi-supervised learning and transfer learning jointly, leading to a more practical and competitive paradigm that can utilize both powerful pre-trained models from source domain as well as labeled/unlabeled data in the target domain. To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization that consists of two complementary components: Adaptive Knowledge Consistency (AKC) on the examples between the source and target model, and Adaptive Representation Consistency (ARC) on the target model between labeled and unlabeled examples. Examples involved in the consistency regularization are adaptively selected according to their potential contributions to the target task. We conduct extensive experiments on popular benchmarks including CIFAR-10, CUB-200, and MURA, by fine-tuning the ImageNet pre-trained ResNet-50 model. Results show that our proposed adaptive consistency regularization outperforms state-of-the-art semi-supervised learning techniques such as Pseudo Label, Mean Teacher, and FixMatch. Moreover, our algorithm is orthogonal to existing methods and thus able to gain additional improvements on top of MixMatch and FixMatch. Our code is available at https://github.com/Walleclipse/Semi-Supervised-Transfer-Learning-Paddle.
Perturbation-Seeking Generative Adversarial Networks: A Defense Framework for Remote Sensing Image Scene Classification

The methods for remote sensing image (RSI) scene classification based on deep convolutional neural networks (DCNNs) have achieved prominent success. However, confronted with adversarial examples obtained by adding imperceptible perturbations to clean images, the great vulnerability of DCNNs makes it worth exploring effective defense methods. To date, numerous countermeasures for adversarial examples have been proposed, but how to improve the defensive ability for unknown attacks still to be answered. To address this issue, in this article, we propose an effective defense framework specified for RSI scene classification, named perturbation-seeking generative adversarial networks (PSGANs). In brief, a new training framework is designed to train the classifier by introducing the examples generated during the image reconstruction process, in addition to clean examples and adversarial ones. These generated examples can be random kinds of unknown attacks during training and thus are utilized to eliminate the blind spots of a classifier. To assist the proposed training framework, a reconstruction method is developed. First, instead of modeling the distribution of clean examples, we model the distributions of the perturbations added in adversarial examples. Second, to make a tradeoff between the diversity of the reconstructed examples and the optimization of PSGAN, a scale factor named seeking radius is introduced to scale the generated perturbations before they are subtracted by the given adversarial examples. Comprehensive and extensive experimental results on three widely used benchmarks for RSI scene classification demonstrate the great effectiveness of PSGAN when faced with both known and unknown attacks. Our source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/xuxiangsun/PSGAN</uri> .
High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network

Existing image-to-image translation (I2IT) methods are either constrained to low-resolution images or long inference time due to their heavy computational burden on the convolution of high-resolution feature maps. In this paper, we focus on speeding-up the high-resolution photorealistic I2IT tasks based on closed-form Laplacian pyramid decomposition and reconstruction. Specifically, we reveal that the attribute transformations, such as illumination and color manipulation, relate more to the low-frequency component, while the content details can be adaptively refined on high-frequency components. We consequently propose a Laplacian Pyramid Translation Network (LPTN) to simultaneously perform these two tasks, where we design a lightweight network for translating the low-frequency component with reduced resolution and a progressive masking strategy to efficiently refine the high-frequency ones. Our model avoids most of the heavy computation consumed by processing high-resolution feature maps and faithfully preserves the image details. Extensive experimental results on various tasks demonstrate that the proposed method can translate 4K images in real-time using one normal GPU while achieving comparable transformation performance against existing methods. Datasets and codes are available: https://github.com/csjliang/LPTN.
From stopping to shopping: An observational study comparing a humanoid service robot with a tablet service kiosk to attract and convert shoppers

This study investigates the effectiveness of a humanoid service robot (HSR) versus a tablet service kiosk (TSK) along the stages of the point-of-sale (POS) conversion funnel. The observational data gathered by means of a field experiment show that the HSR elicited 26 times more interactions (i.e., passersby touching the screen) than the TSK and that these interactions lasted almost +50% as long. Moreover, more people looked at the store and consequently entered it when the HSR was deployed. Furthermore, more unique transactions were registered, and a higher amount was spent during the days when the HSR was present in front of the store. This study proves that implementing an HSR in the store environment is more effective than a TSK in attracting passersby and converting them into buyers.
Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition

In recent years, a number of approaches based on 2D or 3D convolutional neural networks (CNN) have emerged for video action recognition, achieving state-of-the-art results on several large-scale benchmark datasets. In this paper, we carry out in-depth comparative analysis to better understand the differences between these approaches and the progress made by them. To this end, we develop an unified framework for both 2D-CNN and 3D-CNN action models, which enables us to remove bells and whistles and provides a common ground for fair comparison. We then conduct an effort towards a large-scale analysis involving over 300 action recognition models. Our comprehensive analysis reveals that a) a significant leap is made in efficiency for action recognition, but not in accuracy; b) 2D-CNN and 3D-CNN models behave similarly in terms of spatio-temporal representation abilities and transferability. Our codes are available at https://github.com/IBM/action-recognition-pytorch.
Comparison of physical and machine learning models for estimating solar irradiance and photovoltaic power

Conventional models to estimate solar irradiance and photovoltaic power rely on physics and use empirical correlations to handle regional climate and complex physics. Recently, machine learning emerges as an advanced statistical tool to construct more accurate correlations between inputs and outputs. Although machine learning has been applied for modeling solar irradiance and power, no study has reported the accuracy improvement by machine learning compared to conventional physical models. Hence, this study aims to compare the accuracies of physical and machine learning models at each step of solar power modeling, i.e., modeling of global horizontal irradiance, direct normal irradiance, global tilted irradiance, and photovoltaic power. Comparison results demonstrated that machine learning models generally outperform physical models when input parameters are appropriately selected. Machine learning models more significantly reduced the mean bias difference (MBD) than the root mean square difference (RMSD). For global horizontal irradiance and photovoltaic power, machine learning models led to substantially unbiased estimations with 0.96% and 0.03% of MBD, respectively. Among machine learning algorithms, long short-term memory and gated recurrent unit were more recommendable. However, the physical model for solar power estimation was more efficient to reduce RMSD because of their ability to consider constant parameters as input.
A year in lockdown

In March 2020, the World Health Organization declared the Corona Virus 2019 (COVID-19) outbreak a global pandemic. As a result, billions of people were either encouraged or forced by their governments to stay home to reduce the spread of the virus. This caused many to turn to the Internet for work, education, social interaction, and entertainment. With the Internet demand rising at an unprecedented rate, the question of whether the Internet could sustain this additional load emerged. To answer this question, this paper will review the impact of the first year of the COVID-19 pandemic on Internet traffic in order to analyze its performance. In order to keep our study broad, we collect and analyze Internet traffic data from multiple locations at the core and edge of the Internet. From this, we characterize how traffic and application demands change, to describe the "new normal," and explain how the Internet reacted during these unprecedented times.
Amalgamating Knowledge from Heterogeneous Graph Neural Networks

In this paper, we study a novel knowledge transfer task in the domain of graph neural networks (GNNs). We strive to train a multi-talented student GNN, without accessing human annotations, that “amalgamates” knowledge from a couple of teacher GNNs with heterogeneous architectures and handling distinct tasks. The student derived in this way is expected to integrate the expertise from both teachers while maintaining a compact architecture. To this end, we propose an innovative approach to train a slimmable GNN that enables learning from teachers with varying feature dimensions. Meanwhile, to explicitly align topological semantics between the student and teachers, we introduce a topological attribution map (TAM) to highlight the structural saliency in a graph, based on which the student imitates the teachers’ ways of aggregating information from neighbors. Experiments on seven datasets across various tasks, including multi-label classification and joint segmentation-classification, demonstrate that the learned student, with a lightweight architecture, achieves gratifying results on par with and sometimes even superior to those of the teachers in their specializations. Our code is publicly available at https://github.com/ycjing/AmalgamateGNN.PyTorch.
Affective Image Content Analysis: Two Decades Review and New Perspectives

Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges - the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.
Graph-based High-Order Relation Discovery for Fine-grained Recognition

Fine-grained object recognition aims to learn effective features that can identify the subtle differences between visually similar objects. Most of the existing works tend to amplify discriminative part regions with attention mechanisms. Besides its unstable performance under complex backgrounds, the intrinsic interrelationship between different semantic features is less explored. Toward this end, we propose an effective graph-based relation discovery approach to build a contextual understanding of high-order relationships. In our approach, a high-dimensional feature bank is first formed and jointly regularized with semantic- and positional-aware high-order constraints, endowing rich attributes to feature representations. Second, to overcome the high-dimension curse, we propose a graph-based semantic grouping strategy to embed this high-order tensor bank into a low-dimensional space. Meanwhile, a group-wise learning strategy is proposed to regularize the features focusing on the cluster embedding center. With the collaborative learning of three modules, our module is able to grasp the stronger contextual details of fine-grained objects. Experimental evidence demonstrates our approach achieves new state-of-the-art on 4 widely-used fine-grained object recognition benchmarks.
Advances and Opportunities in Remote Sensing Image Geometric Registration: A systematic review of state-of-the-art approaches and future research directions

Geometric registration is often an accuracy assurance for most remote sensing image processing and analysis, such as image mosaicking, image fusion, and time-series analysis. In recent decades, geometric registration has attracted considerable attention in the remote sensing community, leading to a large amount of research on the subject. However, few studies have systematically reviewed its current status and deeply investigated its development trends. Moreover, new approaches are constantly emerging, and some issues still need to be solved. Thus, this article presents a survey of state-of-the-art approaches for remote sensing image registration in terms of intensity-based, feature-based, and combination techniques. Optical flow estimation and deep learning-based methods are summarized, and software-operated registration and registration evaluation are introduced. Building on recent advances, promising opportunities are explored.
IoT with BlockChain: A Futuristic Approach in Agriculture and Food Supply Chain

Agricultural food production is projected to be 70% higher by 2050 than it is today, with the world population rising to more than 9 billion, 34% higher than it is now. The farmers have been forced to produce more with the same resources. This pressure means that optimizing productivity is one of the main objectives of the producers but also in a sustainable way. Not only does agriculture face a decline in production, but it has also had to face limitations in data collection, storing, securing, and sharing, climate change, increases in input prices, traditional food supply chain systems where there is no direct connection between the farmer and the buyer, and limitations on energy use. Existing IoT-based agriculture systems have a centralized format and operate in isolation, leaving room for unresolved issues and major concerns, including data security, manipulation, and single failure points. This paper proposes a futuristic IoT with a blockchain model to meet these challenges. Further, this paper also proposes and novel energy-efficient clustering IoT-based agriculture protocol for lower energy consumption and network stability and compares its results with its counterpart low-energy adoptive clustering hierarchy (LEACH) protocol. The simulation results show that the proposed protocol network stability is 23% higher as compared to LEACH as first node of LEACH dies at 168 rounds while IoT-based agriculture first node dies after 463 rounds. Similarly, IoT-based agriculture protocol energy consumption is 68% lower than that of LEACH. The proposed protocol also extends the network life to more rounds and demonstrates an increase of 112%.
A survey on graph-based deep learning for computational histopathology

With the remarkable success of representation learning for prediction problems, we have witnessed a rapid expansion of the use of machine learning and deep learning for the analysis of digital pathology and biopsy image patches. However, learning over patch-wise features using convolutional neural networks limits the ability of the model to capture global contextual information and comprehensively model tissue composition. The phenotypical and topological distribution of constituent histological entities play a critical role in tissue diagnosis. As such, graph data representations and deep learning have attracted significant attention for encoding tissue representations, and capturing intra- and inter- entity level interactions. In this review, we provide a conceptual grounding for graph analytics in digital pathology, including entity-graph construction and graph architectures, and present their current success for tumor localization and classification, tumor invasion and staging, image retrieval, and survival prediction. We provide an overview of these methods in a systematic manner organized by the graph representation of the input image, scale, and organ on which they operate. We also outline the limitations of existing techniques, and suggest potential future research directions in this domain.
A Straightforward Framework for Video Retrieval Using CLIP

Video Retrieval is a challenging task where the task aims at matching a text query to a video or vice versa. Most of the existing approaches for addressing such a problem rely on annotations made by the users. Although simple, this approach is not always feasible in practice. In this work, we explore the application of the language-image model, CLIP, to obtain video representations without the need for said annotations. This model was explicitly trained to learn a common space where images and text can be compared. Using various techniques described in this document, we extended its application to videos, obtaining state-of-the-art results on the MSR-VTT and MSVD benchmarks.
Data-Uncertainty Guided Multi-Phase Learning for Semi-Supervised Object Detection

In this paper, we delve into semi-supervised object detection where unlabeled images are leveraged to break through the upper bound of fully-supervised object detection. Previous semi-supervised methods based on pseudo labels are severely degenerated by noise and prone to overfit to noisy labels, thus are deficient in learning different unlabeled knowledge well. To address this issue, we propose a data-uncertainty guided multi-phase learning method for semisupervised object detection. We comprehensively consider divergent types of unlabeled images according to their difficulty levels, utilize them in different phases, and ensemble models from different phases together to generate ultimate results. Image uncertainty guided easy data selection and region uncertainty guided RoI Re-weighting are involved in multi-phase learning and enable the detector to concentrate on more certain knowledge. Through extensive experiments on PASCAL VOC and MS COCO, we demonstrate that our method behaves extraordinarily compared to baseline approaches and outperforms them by a large margin, more than 3% on VOC and 2% on COCO.
Multi-label feature selection via manifold regularization and dependence maximization

Feature selection is able to select more discriminative features for classification and plays an important role in multi-label learning to alleviate the effect of the curse of dimensionality. Recently, the multi-label feature selection methods based on the sparse regression model have received increasing attentions. However, most of these methods directly project original data space to label space in the regression model, which is inappropriate because the linear assumption between data space and label space doesn't hold in most cases. In the paper, we propose a feature selection method named multi-label feature selection via manifold regularization and dependence maximization (MRDM). In the regression model of MRDM, the original data space is projected to a low-dimensional manifold space, which not only has the same topological structure with the original data, but also has a strong dependence with the class labels. Then, an objective function involving l2,1-norm regularization is formulated, and an alternating optimization-based iterative algorithm is designed to obtain the sparse coefficients for multi-label feature selection. Extensive experiments on various multi-label data sets demonstrate the superiority of the proposed method compared with some state-of-the-art multi-label feature selection methods.
Artificial intelligence in educational assessment: ‘Breakthrough? Or buncombe and ballyhoo?’

Abstract Artificial Intelligence is at the heart of modern society with computers now capable of making process decisions in many spheres of human activity. In education, there has been intensive growth in systems that make formal and informal learning an anytime, anywhere activity for billions of people through online open educational resources and massive online open courses. Moreover, new developments in Artificial Intelligence‐related educational assessment are attracting increasing interest as means of improving assessment efficacy and validity, with much attention focusing on the analysis of the large volumes of process data being captured from digital assessment contexts. In evaluating the state of play of Artificial Intelligence in formative and summative educational assessment, this paper offers a critical perspective on the two core applications: automated essay scoring systems and computerized adaptive tests, along with the Big Data analysis approaches to machine learning that underpin them.
Density-based weighting for imbalanced regression

Abstract In many real world settings, imbalanced data impedes model performance of learning algorithms, like neural networks, mostly for rare cases. This is especially problematic for tasks focusing on these rare occurrences. For example, when estimating precipitation, extreme rainfall events are scarce but important considering their potential consequences. While there are numerous well studied solutions for classification settings, most of them cannot be applied to regression easily. Of the few solutions for regression tasks, barely any have explored cost-sensitive learning which is known to have advantages compared to sampling-based methods in classification tasks. In this work, we propose a sample weighting approach for imbalanced regression datasets called DenseWeight and a cost-sensitive learning approach for neural network regression with imbalanced data called DenseLoss based on our weighting scheme. DenseWeight weights data points according to their target value rarities through kernel density estimation (KDE). DenseLoss adjusts each data point’s influence on the loss according to DenseWeight, giving rare data points more influence on model training compared to common data points. We show on multiple differently distributed datasets that DenseLoss significantly improves model performance for rare data points through its density-based weighting scheme. Additionally, we compare DenseLoss to the state-of-the-art method SMOGN, finding that our method mostly yields better performance. Our approach provides more control over model training as it enables us to actively decide on the trade-off between focusing on common or rare cases through a single hyperparameter, allowing the training of better models for rare data points.