Reviewerly/exHarmony huggingface dataset:
A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier

Sign language is the comprehensive medium of mass communication for hearing and speaking impaired individuals.As they cannot speak or hear, they are not able to use sound or vocal signals as an information medium for their communication.Rather, they are bound to exchange visual signals to express their feeling in their day-to-day life.For this, they use various body language mainly hand gestures as sign language.Sign language fundamentals can be largely divided into two parts namely digits (numerals) and characters (alphabetical).In this paper, we proposed a hybrid model consisting of a deep transfer learning-based convolutional neural network with a random forest classifier for the automatic recognition of Bangla Sign Language (numerals and alphabets).The overall performance of the presented system is verified on 'Ishara-Bochon' and 'Ishara-Lipi' datasets.'Ishara-Bochon' and 'Ishara-Lipi' are datasets of isolated numerals and alphabets respectively which are the first complete multipurpose open-access dataset for Bangla Sign Language (BSL).Besides, we also proposed a background elimination algorithm that removes unnecessary features from the sign images.Along with the proposed background elimination technique, the system is able to achieve accuracy, precision, recall, f1-score values of 91.67%, 93.64%, 91.67%, 91.47% for character recognition and 97.33%, 97.89%, 97.33%, 97.37% for digit recognition respectively.The detailed experimental analysis assures the feasibility and effectiveness of the proposed system for BSL recognition.
OpenFL: the open federated learning library

Objective.Federated learning (FL) is a computational paradigm that enables organizations to collaborate on machine learning (ML) and deep learning (DL) projects without sharing sensitive data, such as patient records, financial data, or classified secrets.Approach.Open federated learning (OpenFL) framework is an open-source python-based tool for training ML/DL algorithms using the data-private collaborative learning paradigm of FL, irrespective of the use case. OpenFL works with training pipelines built with both TensorFlow and PyTorch, and can be easily extended to other ML and DL frameworks.Main results.In this manuscript, we present OpenFL and summarize its motivation and development characteristics, with the intention of facilitating its application to existing ML/DL model training in a production environment. We further provide recommendations to secure a federation using trusted execution environments to ensure explicit model security and integrity, as well as maintain data confidentiality. Finally, we describe the first real-world healthcare federations that use the OpenFL library, and highlight how it can be applied to other non-healthcare use cases.Significance.The OpenFL library is designed for real world scalability, trusted execution, and also prioritizes easy migration of centralized ML models into a federated training pipeline. Although OpenFL's initial use case was in healthcare, it is applicable beyond this domain and is now reaching wider adoption both in research and production settings. The tool is open-sourced atgithub.com/intel/openfl.
Explainability of artificial intelligence methods, applications and challenges: A comprehensive survey

The continuous advancement of Artificial Intelligence (AI) has been revolutionizing the strategy of decision-making in different life domains. Regardless of this achievement, AI algorithms have been built as Black-Boxes, that is as they hide their internal rationality and learning methodology from the human leaving many unanswered questions about how and why the AI decisions are made. The absence of explanation results in a sensible and ethical challenge. Explainable Artificial Intelligence (XAI) is an evolving subfield of AI that emphasizes developing a plethora of tools and techniques for unboxing the Black-Box AI solutions by generating human-comprehensible, insightful, and transparent explanations of AI decisions. This study begins by discussing the primary principles of XAI research, Black-Box problems, the targeted audience, and the related notion of explainability over the historical timeline of the XAI studies and accordingly establishes an innovative definition of explainability that addresses the earlier theoretical proposals. According to an extensive analysis of the literature, this study contributes to the body of knowledge by driving a fine-grained, multi-level, and multi-dimension taxonomy for insightful categorization of XAI studies with the main aim to shed light on the variations and commonalities of existing algorithms paving the way for extra methodological developments. Then, an experimental comparative analysis is presented for the explanation generated by common XAI algorithms applied to different categories of data to highlight their properties, advantages, and flaws. Followingly, this study discusses and categorizes the evaluation metrics for the XAI-generated explanation and the findings show that there is no common consensus on how an explanation must be expressed, and how its quality and dependability should be evaluated. The findings show that XAI can contribute to realizing responsible and trustworthy AI, however, the advantages of interpretability should be technically demonstrated, and complementary procedures and regulations are required to give actionable information that can empower decision-making in real-world applications. Finally, the tutorial is crowned by discussing the open research questions, challenges, and future directions that serve as a roadmap for the AI community to advance the research in XAI and to inspire specialists and practitioners to take the advantage of XAI in different disciplines.
Image super-resolution: A comprehensive review, recent trends, challenges and applications

Super resolution (SR) is an eminent system in the field of computer vison and image processing to improve the visual perception of the poor-quality images. The key objective of image super resolution is to address the limitations of imaging systems mainly due to hardware problems and requirements for clinical processing of medical imaging using post-processing operations. Numerous super resolution strategies have been put-forward in the computer vision community to improve and achieve high-resolution images over the years. In the past few years, there has been a significant advancement in image super-resolution algorithms. This paper aims to provide the detailed survey on recent advancements in image super-resolution in terms of traditional, deep learning and the latest transformer-based algorithms. The in-depth taxonomy of broadly classified super-resolution techniques within these categories has been broadly discussed. An extensive survey has been carried out on deep learning techniques in terms of parameters, architecture, network complexity, depth, learning rate, framework, optimization, and loss function. Furthermore, we also address some of the significant parameters such as problem definition, evaluation metrics, publicly benchmarks datasets, loss functions and applications. In addition, we have performed an experimental analysis and comparison of various benchmark algorithms on publicly available datasets both qualitively and quantitively. Lastly, we conclude our survey by emphasizing some of the prospective future directions and open issues that the community need to address in the future.
Synthetic data as an enabler for machine learning applications in medicine

Synthetic data generation is the process of using machine learning methods to train a model that captures the patterns in a real dataset. Then new or synthetic data can be generated from that trained model. The synthetic data does not have a one-to-one mapping to the original data or to real patients, and therefore has the potential of privacy preserving properties. There is a growing interest in the application of synthetic data across health and life sciences, but to fully realize the benefits, further education, research, and policy innovation is required. This article summarizes the opportunities and challenges of SDG for health data, and provides directions for how this technology can be leveraged to accelerate data access for secondary purposes.
An Energy-optimized Embedded load balancing using DVFS computing in Cloud Data centers

Task scheduling is a significant challenge in the cloud environment as it affects the network’s performance regarding the workload of the cloud machines. It also directly impacts the consumed energy, therefore the profit of the cloud provider. This paper proposed an algorithm that prioritizes the tasks regarding their execution deadline. We also categorize the physical machines considering their configuration status. Henceforth, the proposed method assigns the jobs to the physical machines with the same priority class close to the user. Furthermore, we reduce the consumed energy of the machines processing the low-priority tasks using the DVFS method. The proposed method migrates the jobs to maintain the workload balance, or if the machines’ class changed according to their scores. We have evaluated and validated the proposed method in the CloudSim library. The simulation results demonstrate that the proposed method optimized energy consumption by 12% and power consumption by 20%.
Shape-Former: Bridging CNN and Transformer via ShapeConv for multimodal image matching

As with any data fusion task, the front-end of the pipeline for image fusion, aiming to collect multitudinous physical properties from multimodal images taken by different types of sensors, requires registering the overlapped content of two images via image matching. In other words, the accuracy of image matching will influence directly the subsequent fusion results. In this work, we propose a hybrid correspondence learning architecture, termed as Shape-Former, which is capable of solving matching problems such as multimodal, and multiview cases. Existing attempts have trouble capturing intricate feature interactions for seeking good correspondence, if the image pairs simultaneously suffer from geometric and radiation distortion. To address this, our key is to take advantage of convolutional neural network (CNN) and Transformer for enhancing structure consensus representation ability. Specifically, we introduce a novel ShapeConv so that CNN and Transformer can be generalized to sparse matches learning. Furthermore, we provide a robust soft estimation of outliers mechanism for filtering the response of outliers before capturing shape features. Finally, we also propose coupling multiple consensus representations to further solve the context conflict problems such as local ambiguity. Experiments with variety of datasets reveal that our Shape-Former outperforms state-of-the-art on multimodal image matching, and shows promising generalization ability to different types of image deformations.
A generalized framework of feature learning enhanced convolutional neural network for pathology-image-oriented cancer diagnosis

In this paper, a feature learning enhanced convolutional neural network (FLE-CNN) is proposed for cancer detection from histopathology images. To build a highly generalized computer-aided diagnosis (CAD) system, an information refinement unit employing depth- and point-wise convolutions is meticulously designed, where a dual-domain attention mechanism is adopted to focus primarily on the important areas. By deploying a residual fusion unit, context information is further integrated to extract highly discriminative features with strong representation ability. Experimental results demonstrate the merits of the proposed FLE-CNN in terms of feature extraction, which has achieved average sensitivity, specificity, precision, accuracy and F1 score of 0.9992, 0.9998, 0.9992, 0.9997 and 0.9992 in a five-class cancer detection task, and in comparison to some other advanced deep learning models, above indicators have been improved by 1.23%, 0.31%, 1.24%, 0.5% and 1.26%, respectively. Moreover, the proposed FLE-CNN provides satisfactory results in three important diagnosis, which further validates that FLE-CNN is a competitive CAD model with high generalization ability.
A survey on federated learning: challenges and applications

Federated learning (FL) is a secure distributed machine learning paradigm that addresses the issue of data silos in building a joint model. Its unique distributed training mode and the advantages of security aggregation mechanism are very suitable for various practical applications with strict privacy requirements. However, with the deployment of FL mode into practical application, some bottlenecks appear in the FL training process, which affects the performance and efficiency of the FL model in practical applications. Therefore, more researchers have paid attention to the challenges of FL and sought for various effective research methods to solve these current bottlenecks. And various research achievements of FL have been made to promote the intelligent development of all application areas with privacy restriction. This paper systematically introduces the current researches in FL from five aspects: the basics knowledge of FL, privacy and security protection mechanisms in FL, communication overhead challenges and heterogeneity problems of FL. Furthermore, we make a comprehensive summary of the research in practical applications and prospect the future research directions of FL.
TMSO-Net: Texture adaptive multi-scale observation for light field image depth estimation

Light field can record the four-dimensional information of light rays, i.e. the position and direction information in which depth information is implied. To improve the depth estimation accuracy, we propose a depth estimation algorithm based on convolutional neural network (CNN). First, a single image super resolution algorithm is adopted to spatially super resolve the sub-aperture images (SAIs). Second, to adapt the texture complexity, the SAIs are partitioned into two regions, i.e., simple texture region and complex texture region, based on the texture analysis of the central SAI. Third, the epipolar plane images (EPIs) in horizontal, vertical, 45 degree diagonal, and 135 degree diagonal directions for both complex and simple texture regions are extracted, and the corresponding EPIs for the simple and complex texture regions are fed into the specified network branches. Finally, a fusion module is designed to generate the depth map. Experimental results show that the quality of the estimated depth maps by the proposed method is better than the state-of-the-art methods in terms of both objective quality and subjective quality. Moreover, the proposed method is more robust to noise.
Investigating metaverse marketing for travel and tourism

The metaverse, defined as an interoperated persistent network of shared virtual environments, engenders perception of holistic presence for audiences through avatar embodiment as well as psychological and physiological immersion in synchronous interaction with other agents and objects. Metaverse tours is a newly emerged phenomenon requiring deepened understanding in the travel and tourism industry. The current study, applying pertinent concepts of enactive cognition and positive psychology, proposes and verifies a theoretical framework. As the research result suggests, concomitant effect exercised by perception of holistic presence containing subdimensions of spatial-presence, social-presence, and self-presence alongside holistic happiness comprising hedonia and eudaimonia serves to reinforce actual visit intention towards a tourism destination in the metaverse context. The current study contributes to explicating an effective avenue conducive to bringing out optimal impact of metaverse tours for travel and tourism marketing.
ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization

Weakly-supervised temporal action localization aims to recognize and localize action segments in untrimmed videos given only video-level action labels for training. Without the boundary information of action segments, existing methods mostly rely on multiple instance learning (MIL), where the predictions of unlabeled instances (i.e., video snippets) are supervised by classifying labeled bags (i.e., untrimmed videos). However, this formulation typically treats snippets in a video as independent instances, ignoring the underlying temporal structures within and across action segments. To address this problem, we propose ASM-Loc, a novel WTAL framework that enables explicit, action-aware segment modeling beyond standard MIL-based methods. Our framework entails three segment-centric components: (i) dynamic segment sampling for compensating the contribution of short actions; (ii) intra- and inter-segment attention for modeling action dynamics and capturing temporal dependencies; (iii) pseudo instance-level supervision for improving action boundary prediction. Furthermore, a multi-step refinement strategy is proposed to progressively improve action proposals along the model training process. Extensive experiments on THUMOS-14 and ActivityNet-v1.3 demonstrate the effectiveness of our approach, establishing new state of the art on both datasets. The code and models are publicly available at https://github.com/boheumd/ASM-Loc.
ReSTR: Convolution-free Referring Image Segmentation Using Transformers

Referring image segmentation is an advanced semantic segmentation task where target is not a predefined class but is described in natural language. Most of existing methods for this task rely heavily on convolutional neural networks, which however have trouble capturing long-range dependencies between entities in the language expression and are not flexible enough for modeling interactions between the two different modalities. To address these issues, we present the first convolution-free model for referring image segmentation using transformers, dubbed ReSTR. Since it extracts features of both modalities through transformer encoders, it can capture long-range dependencies between entities within each modality. Also, ReSTR fuses features of the two modalities by a self-attention encoder, which enables flexible and adaptive interactions between the two modalities in the fusion process. The fused features are fed to a segmentation module, which works adaptively according to the image and language expression in hand. ReSTR is evaluated and compared with previous work on all public benchmarks, where it outperforms all existing models.
Bridging Global Context Interactions for High-Fidelity Image Completion

Bridging global context interactions correctly is important for high-fidelity image completion with large masks. Previous methods attempting this via deep or large receptive field (RF) convolutions cannot escape from the dominance of nearby interactions, which may be inferior. In this paper, we propose to treat image completion as a directionless sequence-to-sequence prediction task, and deploy a transformer to directly capture long-range depen-dence. Crucially, we employ a restrictive CNN with small and non-overlapping RF for weighted token representation, which allows the transformer to explicitly model the long-range visible context relations with equal importance in all layers, without implicitly confounding neighboring tokens when larger RFs are used. To improve appearance consistency between visible and generated regions, a novel attention-aware layer (AAL) is introduced to better exploit distantly related high-frequency features. Overall, extensive experiments demonstrate superior performance compared to state-of-the-art methods on several datasets. Code is available at https://github.com/lyndonzheng/TFill.
Global Matching with Overlapping Attention for Optical Flow Estimation

Optical flow estimation is a fundamental task in computer vision. Recent direct-regression methods using deep neural networks achieve remarkable performance improvement. However, they do not explicitly capture long-term motion correspondences and thus cannot handle large motions effectively. In this paper, inspired by the traditional matching-optimization methods where matching is introduced to handle large displacements before energy-based optimizations, we introduce a simple but effective global matching step before the direct regression and develop a learning-based matching-optimization framework, namely GMFlowNet. In GMFlowNet, global matching is efficiently calculated by applying argmax on 4D cost volumes. Additionally, to improve the matching quality, we propose patch-based overlapping attention to extract large context features. Extensive experiments demonstrate that GM-FlowNet outperforms RAFT, the most popular optimization-only method, by a large margin and achieves state-of-the-art performance on standard benchmarks. Thanks to the matching and overlapping attention, GMFlowNet obtains major improvements on the predictions for textureless regions and large motions. Our code is made publicly available at https://github.com/xiaofeng94/GMFlowNet.
Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time

Implicit neural representations such as Neural Radiance Field (NeRF) have focused mainly on modeling static objects captured under multi-view settings where real-time rendering can be achieved with smart data structures, e.g., PlenOctree. In this paper, we present a novel Fourier PlenOctree (FPO) technique to tackle efficient neural mod-eling and real-time rendering of dynamic scenes captured under the free-view video (FVV) setting. The key idea in our FPO is a novel combination of generalized NeRF, PlenOctree representation, volumetric fusion and Fourier transform. To accelerate FPO construction, we present a novel coarse-to-fine fusion scheme that leverages the gen-eralizable NeRF technique to generate the tree via spatial blending. To tackle dynamic scenes, we tailor the implicit network to model the Fourier coefficients of time-varying density and color attributes. Finally, we construct the FPO and train the Fourier coefficients directly on the leaves of a union PlenOctree structure of the dynamic sequence. We show that the resulting FPO enables compact memory overload to handle dynamic objects and supports efficient fine-tuning. Extensive experiments show that the proposed method is 3000 times faster than the original NeRF and achieves over an order of magnitude acceleration over SOTA while preserving high visual quality for the free-viewpoint rendering of unseen dynamic scenes.
Learning Trajectory-Aware Transformer for Video Super-Resolution

Video super-resolution (VSR) aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Although some progress has been made, there are grand challenges to effectively utilize temporal dependency in entire video sequences. Existing approaches usually align and aggregate video frames from limited adjacent frames (e.g., 5 or 7 frames), which prevents these approaches from satisfactory results. In this paper, we take one step further to enable effective spatio-temporal learning in videos. We propose a novel Trajectory-aware Transformer for Video Super-Resolution (TTVSR). In particular, we formulate video frames into several pre-aligned trajectories which consist of continuous visual tokens. For a query token, self-attention is only learned on relevant visual tokens along spatio-temporal trajectories. Compared with vanilla vision Transformers, such a design significantly reduces the computational cost and enables Transformers to model long-range features. We further propose a cross-scale feature tokenization module to over-come scale-changing problems that often occur in long-range videos. Experimental results demonstrate the superiority of the proposed TTVSR over state-of-the-art models, by extensive quantitative and qualitative evaluations in four widely-used video super-resolution benchmarks. Both code and pre-trained models can be downloaded at https://github.com/researchmm/TTVSR.