Reviewerly/exHarmony huggingface dataset:
Masked face recognition with convolutional neural networks and local binary patterns

Face recognition is one of the most common biometric authentication methods as its feasibility while convenient use. Recently, the COVID-19 pandemic is dramatically spreading throughout the world, which seriously leads to negative impacts on people's health and economy. Wearing masks in public settings is an effective way to prevent viruses from spreading. However, masked face recognition is a highly challenging task due to the lack of facial feature information. In this paper, we propose a method that takes advantage of the combination of deep learning and Local Binary Pattern (LBP) features to recognize the masked face by utilizing RetinaFace, a joint extra-supervised and self-supervised multi-task learning face detector that can deal with various scales of faces, as a fast yet effective encoder. In addition, we extract local binary pattern features from masked face's eye, forehead and eyebow areas and combine them with features learnt from RetinaFace into a unified framework for recognizing masked faces. In addition, we collected a dataset named COMASK20 from 300 subjects at our institution. In the experiment, we compared our proposed system with several state of the art face recognition methods on the published Essex dataset and our self-collected dataset COMASK20. With the recognition results of 87% f1-score on the COMASK20 dataset and 98% f1-score on the Essex dataset, these demonstrated that our proposed system outperforms Dlib and InsightFace, which has shown the effectiveness and suitability of the proposed method. The COMASK20 dataset is available on https://github.com/tuminguyen/COMASK20 for research purposes.
Technology Integration in Higher Education During COVID-19: An Assessment of Online Teaching Competencies Through Technological Pedagogical Content Knowledge Model

The COVID-19 pandemic significantly shifted education from traditional to an online version, which was an emergent state for teachers and students. The substantive situation thus raises the importance of technology integration in education, and teachers are required to update their competencies, respectively. In this regard, the study assessed online teaching competencies of faculty members following, technological pedagogical content knowledge (TPACK) model. Closed-ended surveys were employed for quantitative analysis of randomly selected 256 faculty members from public universities in Karachi, Pakistan. Results indicated that teachers possessed adequate levels of knowledge across all the domains of TPACK. The highest competency was obtained by content knowledge (CK), while technological knowledge (TK) was reported at the lowest level. Furthermore, a significant difference was noted in terms of gender and teaching experience. Correspondingly, the study proposes that the TPACK model should be employed in the professional development programs to develop teachers’ TPACK for integrating information communication and technology in the pedagogical practices. The findings of the study present a constructive overview of teachers’ digital competencies and technology use in teaching and learning in the time of the COVID-19 and also play a significant role in the integration of technology in the post-pandemic time in higher education. The study also suggests relevant educational authorities and policymakers for assessing and enhancing the technological competencies of teachers for quality online education.
Augmented reality in online retailing: a systematic review and research agenda

Purpose Augmented reality (AR) has received massive attention in online retail. Therefore, the paper aims to review the state-of-the-art literature on AR in online retailing, by identifying the antecedents, drives, outcomes, theoretical lenses, typology and methodological approaches. The study further aims to identify the critical avenues for future research. Design/methodology/approach To advance the conceptual and managerial understanding of AR, the study synthesizes the literature through a systematic literature review approach by reviewing 53 articles. Findings Several AR characteristics significantly influence utilitarian, hedonic, perceived risk and experiential value, ultimately resulting in a positive attitude, decision-making assistance and behavioural intentions, wherein customer experience (flow, spatial presence, mental imagery and immersion) plays a mediating role in the process. The study also lists the top authors, articles, journals, countries, theories and methodology used. Originality/value The study provides a comprehensive framework on consumer behaviour towards AR in online retailing. Further, the study proposes the future research agenda in the social side of AR, the dark side of AR, customer engagement, use of AR for experiential value and AR marketing domain.
A temporal distributed hybrid deep learning model for day-ahead distributed PV power forecasting

• A linear components extraction method is proposed to identify the daily fluctuating patterns of the solar power series. • A scenarios generation algorithm is proposed to provide forecasting linear trend data. • Temporal distributed deep learning models are developed based on the detected daily fluctuating patterns of the PV power generation. • The proposed algorithm can provide accurate day-ahead solar power forecasts for distributed PV farms without local weather information. In recent years, the integration of distributed photovoltaic has witnessed explosive growth because of its great economic and environmental benefits. However, the intermittence and randomicity of PV power generation are challenging the stable and economic operation of active distribution system, especially when the local historical and forecasting meteorological data are not available. Thus, this paper presents a novel Gated Recurrent Unit based hybrid model for forecasting distributed PV power generations. The proposed method uses locally historical PV power generation data to forecast one-day-ahead solar power time series with five-minute intervals. First, a linear components extraction method is proposed to identify the daily fluctuating patterns of the solar power series. Then, temporal distributed Gated Recurrent Unit models are developed based on the detected daily fluctuating patterns of the PV power generation. A scenarios generation algorithm is proposed to provide forecasting linear trend data for each Gated Recurrent Unit model. Finally, the daily linear series scenarios, the linear and nonlinear parts of the historical data are fed into the Gated Recurrent Unit to forecast the PV generation. Data collected from an actual distributed PV farm in southeastern China are utilized to validate the effectiveness of the proposed model. The results show that the proposed model can provide a nominalized root mean square error of 6.83% and a nominalized mean absolute error of 4.12%. Compared to a single Gated Recurrent Unit forecasting model, the developed model has improved the forecasting accuracy by 62.7%.
Robust fine-tuning of zero-shot models

Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution. On ImageNet and five derived distribution shifts, WiSE-FT improves accuracy under distribution shift by 4 to 6 percentage points (pp) over prior work while increasing ImageNet accuracy by 1.6 pp. WiSE-FT achieves similarly large robustness gains (2 to 23 pp) on a diverse set of six further distribution shifts, and accuracy gains of 0.8 to 3.3 pp compared to standard fine-tuning on commonly used transfer learning datasets. These improvements come at no additional computational cost during fine-tuning or inference.
Taxonomy of Adaptive Neuro-Fuzzy Inference System in Modern Engineering Sciences

Adaptive Neuro-Fuzzy Inference System (ANFIS) blends advantages of both Artificial Neural Networks (ANNs) and Fuzzy Logic (FL) in a single framework. It provides accelerated learning capacity and adaptive interpretation capabilities to model complex patterns and apprehends nonlinear relationships. ANFIS has been applied and practiced in various domains and provided solutions to commonly recurring problems with improved time and space complexity. Standard ANFIS has certain limitations such as high computational expense, loss of interpretability in larger inputs, curse of dimensionality, and selection of appropriate membership functions. This paper summarizes that the standard ANFIS is unsuitable for complex human tasks that require precise handling of machines and systems. The state-of-the-art and practice research questions have been discussed, which primarily focus on the applicability of ANFIS in the diversifying field of engineering sciences. We conclude that the standard ANFIS architecture is vastly improved when amalgamated with metaheuristic techniques and further moderated with nature-inspired algorithms through calibration and tuning of parameters. It is significant in adapting and automating complex engineering tasks that currently depend on human discretion, prominent in the mechanical, electrical, and geological fields.
FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting

Transformer, as a strong and flexible architecture for modelling long-range relations, has been widely explored in vision tasks. However, when used in video inpainting that requires fine-grained representation, existed method still suffers from yielding blurry edges in detail due to the hard patch splitting. Here we aim to tackle this problem by proposing FuseFormer, a Transformer model designed for video inpainting via fine-grained feature fusion based on novel Soft Split and Soft Composition operations. The soft split divides feature map into many patches with given overlapping interval. On the contrary, the soft composition operates by stitching different patches into a whole feature map where pixels in overlapping regions are summed up. These two modules are first used in tokenization before Transformer layers and de-tokenization after Transformer layers, for effective mapping between tokens and features. Therefore, sub-patch level information interaction is enabled for more effective feature propagation between neighboring patches, resulting in synthesizing vivid content for hole regions in videos. Moreover, in FuseFormer, we elaborately insert the soft composition and soft split into the feed-forward network, enabling the 1D linear layers to have the capability of modelling 2D structure. And, the sub-patch level feature fusion ability is further enhanced. In both quantitative and qualitative evaluations, our proposed FuseFormer surpasses state-of-the-art methods. We also conduct detailed analysis to examine its superiority. Code and pretrained models are available at https://github.com/ruiliu-ai/FuseFormer.
GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition

In recent years, the growing utilization of medical imaging is placing an increasing burden on radiologists. Deep learning provides a promising solution for automatic medical image analysis and clinical decision support. However, large-scale manually labeled datasets required for training deep neural networks are difficult and expensive to obtain for medical images. The purpose of this work is to develop label-efficient multimodal medical imaging representations by leveraging radiology reports. We propose an attention-based framework for learning global and local representations by contrasting image sub-regions and words in the paired report. In addition, we propose methods to leverage the learned representations for various downstream medical image recognition tasks with limited labels. Our results demonstrate high-performance and label-efficiency for image-text retrieval, classification (finetuning and zerosshot settings), and segmentation on different datasets.
MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis

In this paper, we propose MINE to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our approach is a continuous depth generalization of the Multiplane Images (MPI) by introducing the NEural radiance fields (NeRF). Given a single image as input, MINE predicts a 4-channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our MINE outperforms state-of-the-art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision. Our source code is available at https://github.com/vincentfung13/MINE.
Syncretic Modality Collaborative Learning for Visible Infrared Person Re-Identification

Visible infrared person re-identification (VI-REID) aims to match pedestrian images between the daytime visible and nighttime infrared camera views. The large cross-modality discrepancies have become the bottleneck which limits the performance of VI-REID. Existing methods mainly focus on capturing cross-modality sharable representations by learning an identity classifier. However, the heterogeneous pedestrian images taken by different spectrum cameras differ significantly in image styles, resulting in inferior discriminability of feature representations. To alleviate the above problem, this paper explores the correlation between two modalities and proposes a novel syncretic modality collaborative learning (SMCL) model to bridge the cross-modality gap. A new modality that incorporates features of heterogeneous images is constructed automatically to steer the generation of modality-invariant representations. Challenge enhanced homogeneity learning (CEHL) and auxiliary distributional similarity learning (ADSL) are integrated to project heterogeneous features on a unified space and enlarge the inter-class disparity, thus strengthening the discriminative power. Extensive experiments on two cross-modality benchmarks demonstrate the effectiveness and superiority of the proposed method. Especially, on SYSU-MM01 dataset, our SMCL model achieves 67.39% rank-1 accuracy and 61.78% mAP, surpassing the cutting-edge works by a large margin.
Self-supervised Video Object Segmentation by Motion Grouping

Animals have evolved highly functional visual systems to understand motion, assisting perception even under complex environments. In this paper, we work towards developing a computer vision system able to segment objects by exploiting motion cues, i.e. motion segmentation. To achieve this, we introduce a simple variant of the Transformer to segment optical flow frames into primary objects and the background, which can be trained in a self-supervised manner, i.e. without using any manual annotations. Despite using only optical flow, and no appearance information, as input, our approach achieves superior results compared to previous state-of-the-art self-supervised methods on public benchmarks (DAVIS2016, SegTrackv2, FBMS59), while being an order of magnitude faster. On a challenging camouflage dataset (MoCA), we significantly outperform other self-supervised approaches, and are competitive with the top supervised approach, highlighting the importance of motion cues and the potential bias towards appearance in existing video segmentation models.
Role of machine learning and deep learning in securing 5G-driven industrial IoT applications

The Internet of Things (IoT) connects millions of computing devices and has set a stage for future technology where industrial use cases like smart cities and smart houses will operate with minimal human intervention. IoT’s cross-domain amalgamations with emergent technologies like 5G and blockchain affects human life. Hence, increase in reliance over IoT necessitates focus on its privacy and security concerns. Implementing security through encryption, authentication, access control and communication security is the need of the hour. These needs can be best catered with the use of machine learning (ML) and deep learning (DL) that can help in realizing secure intelligent systems. In this work, the authors present a comprehensive review for securing Industrial-IoT (I-IoT) devices to contribute to the development of security methods for I-IoT deployed over 5G and blockchain. The survey provides a general analysis of the state-of-the-art security implementation and further assesses the product life cycle of IoT devices. The authors present numerous virtues as well as faults in the machine learning and deep learning algorithms deployed over the fog architecture in context with the security solutions. The potential security algorithms can help overcome many challenges in the IoT security and pave way for implementation with emerging technologies like 5G, blockchain, edge computing, fog computing and their use cases for creating smart environments.
High-Order Interaction Learning for Image Captioning

Image captioning aims at understanding various semantic concepts (e.g., objects and relationships) from an image and integrating them in a sentence-level description. Hence, it is necessary to learn the interaction among these concepts. If we define the context of the interaction to be involved in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">subject-predicate-object</i> triplet, most current methods only focus on the single triplet for the first-order interaction to generate sentences. Intuitively, we humans are able to perceive the high-order interaction among concepts from two or more triplets to describe an image. For example, when we see the triplets <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">man-cutting-sandwich</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">man-with-knife</i> , it is natural to integrate and predict the sentence <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">man cutting sandwich with knife</i> . This depends on the high-order interaction between <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cutting</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">knife</i> in different triplets. Therefore, exploiting high-order interaction is expected to benefit image captioning and focus on reasoning. In this paper, we introduce the novel high-order interaction learning method over detected objects and relationships for image captioning under the umbrella of the encoder-decoder framework. We first extract a set of object and relationship features in an image. During the encoding stage, the interactive refining network is proposed to learn high-order representations by modeling intra- and inter-object feature interaction in the self-attention fashion. During the decoding stage, the interactive fusion network is proposed to integrate object and relationship information by strengthening their high-order interaction based on language context for sentence generation. In this way, we learn the object-relationship dependencies in different stages, which can provide abundant cues for both visual understanding and caption generation. Extensive experiments show that the proposed method can achieve competitive performances against the state-of-the-art methods on MSCOCO dataset. Additional ablation studies further validate its effectiveness.
From Synthetic to Real: Image Dehazing Collaborating with Unlabeled Real Data

Single image dehazing is a challenging task, for which the domain shift between synthetic training data and real-world testing images usually leads to degradation of existing methods. To address this issue, we propose a novel image dehazing framework collaborating with unlabeled real data. First, we develop a disentangled image dehazing network (DID-Net), which disentangles the feature representations into three component maps, i.e. the latent haze-free image, the transmission map, and the global atmospheric light estimate, respecting the physical model of a haze process. Our DID-Net predicts the three component maps by progressively integrating features across scales, and refines each map by passing an independent refinement network. Then a disentangled-consistency mean-teacher network (DMT-Net) is employed to collaborate unlabeled real data for boosting single image dehazing. Specifically, we encourage the coarse predictions and refinements of each disentangled component to be consistent between the student and teacher networks by using a consistency loss on unlabeled real data. We make comparison with 13 state-of-the-art dehazing methods on a new collected dataset (Haze4K) and two widely-used dehazing datasets (i.e., SOTS and HazeRD), as well as on real-world hazy images. Experimental results demonstrate that our method has obvious quantitative and qualitative improvements over the existing methods.
ECAP-YOLO: Efficient Channel Attention Pyramid YOLO for Small Object Detection in Aerial Image

Detection of small targets in aerial images is still a difficult problem due to the low resolution and background-like targets. With the recent development of object detection technology, efficient and high-performance detector techniques have been developed. Among them, the YOLO series is a representative method of object detection that is light and has good performance. In this paper, we propose a method to improve the performance of small target detection in aerial images by modifying YOLOv5. The backbone is was modified by applying the first efficient channel attention module, and the channel attention pyramid method was proposed. We propose an efficient channel attention pyramid YOLO (ECAP-YOLO). Second, in order to optimize the detection of small objects, we eliminated the module for detecting large objects and added a detect layer to find smaller objects, reducing the computing power used for detecting small targets and improving the detection rate. Finally, we use transposed convolution instead of upsampling. Comparing the method proposed in this paper to the original YOLOv5, the performance improvement for the mAP was 6.9% when using the VEDAI dataset, 5.4% when detecting small cars in the xView dataset, 2.7% when detecting small vehicle and small ship classes from the DOTA dataset, and approximately 2.4% when finding small cars in the Arirang dataset.