Reviewerly/exHarmony huggingface dataset:
Gloss Semantic-Enhanced Network with Online Back-Translation for Sign Language Production

Sign Language Production (SLP) aims to generate the visual appearance of sign language according to the spoken language, in which a key procedure is to translate sign Gloss to Pose (G2P). Existing G2P methods mainly focus on regression prediction of posture coordinates, namely closely fitting the ground truth. In this paper, we provide a new viewpoint: a Gloss semantic-Enhanced Network is proposed with Online Back-Translation (GEN-OBT) for G2P in the SLP task. Specifically, GEN-OBT consists of a gloss encoder, a pose decoder, and an online reverse gloss decoder. In the gloss encoder based on the transformer, we design a learnable gloss token without any prior knowledge of gloss, to explore the global contextual dependency of the entire gloss sequence. During sign pose generation, the gloss token is aggregated onto the existing generated poses as gloss guidance. Then, the aggregated features are interacted with the entire gloss embedding vectors to generate the next pose. Furthermore, we design a CTC-based reverse decoder to convert the generated poses backward into glosses, which guarantees the semantic consistency during the processes of gloss-to-pose and pose-to-gloss. Extensive experiments on the challenging PHOENIX14T benchmark demonstrate that the proposed GEN-OBT outperforms the state-of-the-art models. Visualization results further validate the interpretability of our method.
Cross-Modality High-Frequency Transformer for MR Image Super-Resolution

Improving the resolution of magnetic resonance (MR) image data is critical to computer-aided diagnosis and brain function analysis. Higher resolution helps to capture more detailed content, but typically induces to lower signal-to-noise ratio and longer scanning time. To this end, MR image super-resolution has become a widely-interested topic in recent times. Existing works establish extensive deep models with the conventional architectures based on convolutional neural networks (CNN). In this work, to further advance this research field, we make an early effort to build a Transformer-based MR image super-resolution framework, with careful designs on exploring valuable domain prior knowledge. Specifically, we consider two-fold domain priors including the high-frequency structure prior and the inter-modality context prior, and establish a novel Transformer architecture, called Cross-modality high-frequency Transformer (Cohf-T), to introduce such priors into super-resolving the low-resolution (LR) MR images. Experiments on two datasets indicate that Cohf-T achieves new state-of-the-art performance.
Multigranular Visual-Semantic Embedding for Cloth-Changing Person Re-identification

To date, only a few works have focused on the cloth-changing person Re-identification (ReID) task, but since it is very difficult to extract generalized and robust features for representing people with different clothes, thus, their performances need to be improved. Moreover, visual-semantic information is also often ignored. To solve these issues, in this work, a novel multigranular visual-semantic embedding algorithm (MVSE) is proposed for cloth-changing person ReID, where visual semantic information and human attributes are embedded into the network, and the generalized features of human appearance can be well learned to effectively solve the problem of cloth-changing. Specifically, to fully represent a person with clothing changes, a multigranular feature representation scheme (MGR) is employed to adaptively extract multilevel and multigranular feature information, and then a cloth desensitization network (CDN) is designed to improve the feature robustness for the person with different clothes, where different high-level human attributes are fully utilized. Moreover, to further solve the issue of pose changes and occlusion under different camera perspectives, a partially semantically aligned network (PSA) is proposed to obtain the visual-semantic information that is used to align the human attributes. Most importantly, these three modules are jointly explored in a unified framework. Extensive experimental results on four cloth-changing person ReID datasets demonstrate that the MVSE algorithm can extract highly robust feature representations of cloth-changing persons, and it can outperform state-of-the-art cloth-changing person ReID approaches.
Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge

Medical vision-and-language pre-training (Med-VLP) has received considerable attention owing to its applicability to extracting generic vision-and-language representations from medical images and texts. Most existing methods mainly contain three elements: uni-modal encoders (i.e., a vision encoder and a language encoder), a multi-modal fusion module, and pretext tasks, with few studies considering the importance of medical domain expert knowledge and explicitly exploiting such knowledge to facilitate Med-VLP. Although there exist knowledge-enhanced vision-and-language pre-training (VLP) methods in the general domain, most require off-the-shelf toolkits (e.g., object detectors and scene graph parsers), which are unavailable in the medical domain. In this paper, we propose a systematic and effective approach to enhance Med-VLP by structured medical knowledge from three perspectives. First, considering knowledge can be regarded as the intermediate medium between vision and language, we align the representations of the vision encoder and the language encoder through knowledge. Second, we inject knowledge into the multi-modal fusion model to enable the model to perform reasoning using knowledge as the supplementation of the input image and text. Third, we guide the model to put emphasis on the most critical information in images and texts by designing knowledge-induced pretext tasks. To perform a comprehensive evaluation and facilitate further research, we construct a medical vision-and-language benchmark including three tasks. Experimental results illustrate the effectiveness of our approach, where state-of-the-art performance is achieved on all downstream tasks. Further analyses explore the effects of different components of our approach and various settings of pre-training.
Target-Driven Structured Transformer Planner for Vision-Language Navigation

Vision-language navigation is the task of directing an embodied agent to navigate in 3D scenes with natural language instructions. For the agent, inferring the long-term navigation target from visual-linguistic clues is crucial for reliable path planning, which, however, has rarely been studied before in literature. In this article, we propose a Target-Driven Structured Transformer Planner (TD-STP) for long-horizon goal-guided and room layout-aware navigation. Specifically, we devise an Imaginary Scene Tokenization mechanism for explicit estimation of the long-term target (even located in unexplored environments). In addition, we design a Structured Transformer Planner which elegantly incorporates the explored room layout into a neural attention architecture for structured and global planning. Experimental results demonstrate that our TD-STP substantially improves previous best methods' success rate by 2% and 5% on the test set of R2R and REVERIE benchmarks, respectively. Our code is available at https://github.com/YushengZhao/TD-STP.
Automatic frequency-based feature selection using discrete weighted evolution strategy

High dimensional datasets usually suffer from curse of dimensionality which may increase the classification time and decrease the classification accuracy beyond a certain dimensionality. Thus, feature selection is used to discard redundant features for improving classification. Nonetheless, there is not a single feature selection method which could deal with all datasets. Thus, this paper proposes an automatic hybrid feature selection incorporating both filter and wrapper methods called Extended Mutual Congestion-Discrete Weighted Evolution Strategy (EMC-DWES). First, Extended Mutual Congestion (EMC) is proposed as a frequency-based filter ranker to discard irrelevant and redundant features using intrinsic statistics of features. Second, Discrete Weighted Evolution Strategy (DWES) is applied on the remaining features selected by EMC to perform the final automatic feature selection within a wrapper method. DWES clusters the features and applies mutation both to select the most relevant feature in each cluster at a time and to avoid selecting redundant features simultaneously through assigning greater weights to most informative clusters. The performance of EMC-DWES (in maximizing classification accuracy and minimizing the selected subset length) is investigated using benchmark high dimensional medical datasets including Covid-19. Likewise, the superiority of EMC-DWES in comparison with state-of-the-art is also evaluated in all datasets. The implementation of EMC-DWES is available on https://github.com/KhaosResearch/EMC-DWES.
You Only Align Once: Bidirectional Interaction for Spatial-Temporal Video Super-Resolution

Spatial-Temporal Video Super-Resolution (ST-VSR) technology generates high-quality videos with higher resolution and higher frame rates. Existing advanced methods accomplish ST-VSR tasks through the association of Spatial and Temporal video super-resolution (S-VSR and T-VSR). These methods require two alignments and fusions in S-VSR and T-VSR, which is obviously redundant and fails to sufficiently explore the information flow of consecutive spatial LR frames. Although bidirectional learning (future-to-past and past-to-future) was introduced to cover all input frames, the direct fusion of final predictions fails to sufficiently exploit intrinsic correlations of bidirectional motion learning and spatial information from all frames. We propose an effective yet efficient recurrent network with bidirectional interaction for ST-VSR, where only one alignment and fusion is needed. Specifically, it first performs backward inference from future to past, and then follows forward inference to super-resolve intermediate frames. The backward and forward inferences are assigned to learn structures and details to simplify the learning task with joint optimizations. Furthermore, a Hybrid Fusion Module (HFM) is designed to aggregate and distill information to refine spatial information and reconstruct high-quality video frames. Extensive experiments on two public datasets demonstrate that our method outperforms state-of-the-art methods in efficiency, and reduces calculation cost by about 22%.
Multimodal Hate Speech Detection via Cross-Domain Knowledge Transfer

Nowadays, the hate speech diffusion of texts and images in social network has become the mainstream compared with the diffusion of texts-only, raising the pressing needs of multimodal hate speech detection task. Current research on this task mainly focuses on the construction of multimodal models without considering the influence of the unbalanced and widely distributed samples for various attacks in hate speech. In this situation, introducing enhanced knowledge is necessary for understanding the attack category of hate speech comprehensively. Due to the high correlation between hate speech detection and sarcasm detection tasks, this paper makes an initial attempt of common knowledge transfer based on the above two tasks, where hate speech detection and sarcasm detection are defined as primary and auxiliary tasks, respectively. A scalable cross-domain knowledge transfer (CDKT) framework is proposed, where the mainstream vision-language transformer could be employed as backbone flexibly. Three modules are included, bridging the semantic, definition and domain gaps simultaneously between primary and auxiliary tasks. Specifically, semantic adaptation module formulates the irrelevant parts between image and text in primary and auxiliary tasks, and disentangles with the text representation to align the visual and word tokens. Definition adaptation module assigns different weights to the training samples of auxiliary task by measuring the correlation between samples of the auxiliary and primary task. Domain adaptation module minimizes the feature distribution gap of samples in two tasks. Extensive experiments show that the proposed CDKT provides a stable improvement compared with baselines and produces a competitive performance compared with some existing multimodal hate speech detection methods.
BadHash: Invisible Backdoor Attacks against Deep Hashing with Clean Label

Due to its powerful feature learning capability and high efficiency, deep hashing has achieved great success in large-scale image retrieval. Meanwhile, extensive works have demonstrated that deep neural networks (DNNs) are susceptible to adversarial examples, and exploring adversarial attack against deep hashing has attracted many research efforts. Nevertheless, backdoor attack, another famous threat to DNNs, has not been studied for deep hashing yet. Although various backdoor attacks have been proposed in the field of image classification, existing approaches failed to realize a truly imperceptive backdoor attack that enjoys invisible triggers and clean label setting simultaneously, and they cannot meet the intrinsic demand of image retrieval backdoor.
Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitals

Abstract Objective Federated learning (FL) allows multiple distributed data holders to collaboratively learn a shared model without data sharing. However, individual health system data are heterogeneous. “Personalized” FL variations have been developed to counter data heterogeneity, but few have been evaluated using real-world healthcare data. The purpose of this study is to investigate the performance of a single-site versus a 3-client federated model using a previously described Coronavirus Disease 19 (COVID-19) diagnostic model. Additionally, to investigate the effect of system heterogeneity, we evaluate the performance of 4 FL variations. Materials and methods We leverage a FL healthcare collaborative including data from 5 international healthcare systems (US and Europe) encompassing 42 hospitals. We implemented a COVID-19 computer vision diagnosis system using the Federated Averaging (FedAvg) algorithm implemented on Clara Train SDK 4.0. To study the effect of data heterogeneity, training data was pooled from 3 systems locally and federation was simulated. We compared a centralized/pooled model, versus FedAvg, and 3 personalized FL variations (FedProx, FedBN, and FedAMP). Results We observed comparable model performance with respect to internal validation (local model: AUROC 0.94 vs FedAvg: 0.95, P = .5) and improved model generalizability with the FedAvg model (P &amp;lt; .05). When investigating the effects of model heterogeneity, we observed poor performance with FedAvg on internal validation as compared to personalized FL algorithms. FedAvg did have improved generalizability compared to personalized FL algorithms. On average, FedBN had the best rank performance on internal and external validation. Conclusion FedAvg can significantly improve the generalization of the model compared to other personalization FL algorithms; however, at the cost of poor internal validity. Personalized FL may offer an opportunity to develop both internal and externally validated algorithms.
A Modified Reptile Search Algorithm for Numerical Optimization Problems

The reptile search algorithm (RSA) is a swarm-based metaheuristic algorithm inspired by the encirclement and hunt mechanisms of crocodiles. Compared with other algorithms, RSA is competitive but still suffers from low population diversity, unbalanced exploitation and exploration, and the tendency to fall into local optima. To overcome these shortcomings, a modified variant of RSA, named MRSA, is proposed in this paper. First, an adaptive chaotic reverse learning strategy is employed to enhance the population diversity. Second, an elite alternative pooling strategy is proposed to balance exploitation and exploration. Finally, a shifted distribution estimation strategy is used to correct the evolutionary direction and improve the algorithm performance. Subsequently, the superiority of MRSA is verified using 23 benchmark functions, IEEE CEC2017 benchmark functions, and robot path planning problems. The Friedman test, the Wilcoxon signed-rank test, and simulation results show that the proposed MRSA outperforms other comparative algorithms in terms of convergence accuracy, convergence speed, and stability.
An Efficient Machine Learning-Based Model to Effectively Classify the Type of Noises in QR Code: A Hybrid Approach

Granting smart device consumers with information, simply and quickly, is what drives quick response (QR) codes and mobile marketing to go hand in hand. It boosts marketing campaigns and objectives and allows one to approach, engage, influence, and transform a wider target audience by connecting from offline to online platforms. However, restricted printing technology and flexibility in surfaces introduce noise while printing QR code images. Moreover, noise is often unavoidable during the gathering and transmission of digital images. Therefore, this paper proposed an automatic and accurate noise detector to identify the type of noise present in QR code images. For this, the paper first generates a new dataset comprising 10,000 original QR code images of varying sizes and later introduces several noises, including salt and pepper, pepper, speckle, Poisson, salt, local var, and Gaussian to form a dataset of 80,000 images. We perform extensive experiments by reshaping the generated images to uniform size for exploiting Convolutional Neural Network (CNN), Support Vector Machine (SVM), and Logistic Regression (LG) to classify the original and noisy images. Later, the analysis is further widened by incorporating histogram density analysis to trace and target highly important features by transforming images of varying sizes to obtain 256 features, followed by SVM, LG, and Artificial Neural Network (ANN) to identify the noise type. Moreover, to understand the impact of symmetry of noises in QR code images, we trained the models with combinations of 3-, 5-, and 7-noise types and analyzed the classification performance. From comparative analyses, it is noted that the Gaussian and Localvar noises possess symmetrical characteristics, as all the classifiers did not perform well to segregate these two noises. The results prove that histogram analysis significantly improves classification accuracy with all exploited models, especially when combined with SVM, it achieved maximum accuracy for 4- and 6-class classification problems.
Metaverse: Research Based Analysis and Impact on Economy and Business

This paper addresses the topic of how the Metaverse can impact the business models of the companies and growth of their economies. A detailed analysis is conducted on four Metaverse cryptocurrencies namely, AXS, MANA, SAND and ILV using the ARIMA and SARIMAX prediction model. Analysis has been done on around 1,460 observations in the data-set that ranges from March 2021 to March 2022. Comparison has been done between the predicted price and the actual closing price for the respective crypto stock prices of the companies. In addition, the analysis also provides the accuracy of predicted values of closing price with a graph visualizer using testing data. The results are an indicator for the potential growth of the respective company’s cryptocurrency token, which in turn is proportional to the growth of the company in the market. Thus, as per our analysis and prediction model, it can be concluded that increased percentage of investments in business models aiming to adapt Metaverse technology would be profitable in the near future and is not just a consequence of Metaverse being a hot topic.
Validating a membership disclosure metric for synthetic health data

One of the increasingly accepted methods to evaluate the privacy of synthetic data is by measuring the risk of membership disclosure. This is a measure of the F1 accuracy that an adversary would correctly ascertain that a target individual from the same population as the real data is in the dataset used to train the generative model, and is commonly estimated using a data partitioning methodology with a 0.5 partitioning parameter.Validate the membership disclosure F1 score, evaluate and improve the parametrization of the partitioning method, and provide a benchmark for its interpretation.We performed a simulated membership disclosure attack on 4 population datasets: an Ontario COVID-19 dataset, a state hospital discharge dataset, a national health survey, and an international COVID-19 behavioral survey. Two generative methods were evaluated: sequential synthesis and a generative adversarial network. A theoretical analysis and a simulation were used to determine the correct partitioning parameter that would give the same F1 score as a ground truth simulated membership disclosure attack.The default 0.5 parameter can give quite inaccurate membership disclosure values. The proportion of records from the training dataset in the attack dataset must be equal to the sampling fraction of the real dataset from the population. The approach is demonstrated on 7 clinical trial datasets.Our proposed parameterization, as well as interpretation and generative model training guidance provide a theoretically and empirically grounded basis for evaluating and managing membership disclosure risk for synthetic data.
Health-zkIDM: A Healthcare Identity System Based on Fabric Blockchain and Zero-Knowledge Proof

The issue of identity authentication for online medical services has been one of the key focuses of the healthcare industry in recent years. Most healthcare organizations use centralized identity management systems (IDMs), which not only limit the interoperability of patient identities between institutions of healthcare, but also create isolation between data islands. The more important matter is that centralized IDMs may lead to privacy disclosure. Therefore, we propose Health-zkIDM, a decentralized identity authentication system based on zero-knowledge proof and blockchain technology, which allows patients to identify and verify their identities transparently and safely in different health fields and promotes the interaction between IDM providers and patients. The users in Health-zkIDM are uniquely identified by one ID registered. The zero-knowledge proof technology is deployed on the client, which provides the user with a proof of identity information and automatically verifies the user's identity after registration. We implemented chaincodes on the Fabric, including the upload of proof of identity information, identification, and verification functions. The experiences show that the performance of the Health-zkIDM system can achieve throughputs higher than 400 TPS in Caliper.
Virtually secure: A taxonomic assessment of cybersecurity challenges in virtual reality environments

Although Virtual Reality (VR) is certainly not a new technology, its recent adoption across several sectors beyond entertainment has led the information security research community to take note of the new cyber threats that come with it. The variety of system components presents an extensive attack surface that can be exploited. At the same time, VR's emphasis on immersion, interaction and presence means that the user can be targeted directly, yet the use of head-mounted displays may prevent them from observing a cyber attack's impact in their immediate physical environment. This paper presents the first taxonomic representation of VR security challenges. By systemically classifying existing VR cyber threats against existing defences in a single comparative matrix, we aim to help researchers from different backgrounds to identify key focus areas where further research would be most beneficial.