Reviewerly/exHarmony huggingface dataset:
Large-field Gesture Tracking and Recognition for Augmented Reality Interaction

Abstract In recent years, with the continuous development of computer vision and artificial intelligence technology, gesture recognition is widely used in many fields, such as virtual reality, augmented reality and so on. However, the traditional binocular camera architecture is limited by its limited field of view Angle and depth perception range. Fisheye camera is gradually applied in gesture recognition field because of its advantage of larger field of view Angle. Fisheye cameras offer a wider field of vision than previous binocular cameras, allowing for a greater range of gesture recognition. This gives fisheye cameras a distinct advantage in situations that require a wide field of view. However, because the imaging mode of fisheye camera is different from traditional camera, the image of fisheye camera has a certain degree of distortion, which makes the calculation of gesture recognition more complicated. Our goal is to design a distortion correction processing strategy suitable for fisheye cameras in order to extend the range of gesture recognition and achieve large field of view gesture recognition. Combined with binocular technology, we can use the acquired hand depth information to enrich the means of interaction. By taking advantage of the large viewing Angle of the fisheye camera to expand the range of gesture recognition, make it more extensive and accurate. This will help improve the real-time and precision of gesture recognition, which has important implications for artificial intelligence, virtual reality and augmented reality.
A modified deep learning based MIMO communication for integrated sensing, communication and computing systems

The future wireless networks are expected to be data-driven integrated sensing, communication and computing (ISCC) systems and benefit from employing deep learning to process the large amount of data with less resources and computational limit. As a result, deep learning based learning MIMO communication has been widely discussed in recent years and show fascinating performance advantages over human designed MIMO systems. However, we find that the autoencoder frameworks proposed in current deep learning based methods are not able to be implemented under multipath channels, which is the most common channel model in actual environment. To overcome this problem, a modified autoencoder framework is proposed in this paper. This proposed autoencoder framework introduces well designed convolutional neural layers into the widely used feedforward neural networks, and demonstrates significant performance improvements over traditional Turbo code method adopted in existing wireless MIMO systems. Moreover, computer simulations are performed and the proposed autoencoder framework is also demonstrated to be effective in SISO and MISO systems under AWGN and Rayleigh fading channels, which verifies the generality of the proposed autoencoder framework for end-to-end learning communication systems.
MISPSO-Attack: An efficient adversarial watermarking attack based on multiple initial solution particle swarm optimization

The vulnerability of deep learning models to adversarial attacks is a growing concern, as the emergence of adversarial samples exposes almost all models to the risk of such attacks. This paper proposes a new method for adversarial attacks through watermarking. Our goal is to leverage the properties of adversarial samples to prevent people’s images from being maliciously collected and compared, thereby avoiding the leakage of private information. Our method, which improves on the multi-swarm particle swarm optimization (MPSO) algorithm, outperforms existing similar methods on two popular computer vision datasets. We conducted attack experiments on the widely used Imagenet dataset and achieve the highest attack success rate of 89.50%. The experimental results demonstrate the superiority of our method over existing similar methods. We simulate the attacks on the online social environment using two face photographs datasets and face recognition models. Our method reaches the best deception performance compared to similar methods, with the highest success rate of 97.03%, demonstrating our approach’s ability to protect individuals’ privacy. Furthermore, we investigate the natural causes of adversarial samples and demonstrate their inevitability, providing valuable insights for developing more robust deep models. The source code of the proposed method is available online at: https://github.com/grandwang/main_attack.
On the outdatedness of workflows in the GitHub Actions ecosystem

GitHub Actions was introduced as a way to automate CI/CD workflows in GitHub, the largest social coding platform. Thanks to its deep integration into GitHub, GitHub Actions can be used to automate a wide range of social and technical activities. Among its main features, it allows automation workflows to rely on reusable components – the so-called Actions – to enable developers to focus on the tasks that should be automated rather than on how to automate them. As any other kind of reusable software components, Actions are continuously updated, causing many automation workflows to use outdated versions of these Actions. Based on a dataset of nearly one million workflows obtained from 22K+ repositories between November 2019 and September 2022, we provide quantitative empirical evidence that reusing Actions in GitHub workflows is common practice, even if this reuse tends to concentrate on a limited number of Actions. We show that Actions are frequently updated, and we quantify to which extent automation workflows are outdated with respect to these Actions. Using two complementary metrics, technical lag and opportunity lag, we found that most of the workflows are using an outdated Action release, are lagging behind the latest available release for at least 7 months, and had the opportunity to be updated during at least 9 months. This calls for a more rigorous management of Action outdatedness in automation workflows, as well as for better policies and tooling to keep workflows up-to-date.
A new paradigm for high-capacity reversible data hiding with pixel repetition and adaptive embedding

Reversible data hiding (RDH) offers a useful attribute of retaining the visual semantics with security properties, for the effective management and protection of digital images with their metadata. This paper introduces a Reversible Data Hiding (RDH) scheme with Pixel Repetition (PR) to selectively embed additional secret data. bits utilizing the image redundancy created with PR. A generalized framework of PR is first developed and the rate–distortion criterion is extended for adaptive embedding. An input image is defined to be scaled-up by replacing each original pixel (called seed) with a block of shape s×s containing the seed, where s is the scaling factor. For better utilizing this PR-based image redundancy, we then propose an RDH scheme with two-phase embedding for s=2. The first phase of the scheme embeds 4-bit data in each 2 × 2 image block using a mapping function, which is designed to modify the repeated pixels, keeping the seed intact in each block. Being invertible, the mapping function also allows another phase of embedding in the preserved seeds using an adaptive selection of their least significant bits. Compared to the prominent PR-based and other related schemes, an impressive rate–distortion performance of the proposed scheme is recorded with a greater potential for the applications that require a high and varying embedding rate with better image quality.
Named Entity Recognition of Electronic Medical Records based on BERT-BiLSTM-Biaffine Model

Abstract In the case of the specific task of identifying named entities within electronic medical record, it is hard to determine the boundary of nested entities, and existing NER systems have insufficient decoding performance. Based on the pre training model BERT, this paper introduces a novel network structure called Biaffine Layer using a bidirectional LSTM layer. The network uses a dual affine attention mechanism for semantic information learning, which can better interact with the semantic information of entity heads and entity tails, thereby achieving better recognition results for entities. Due to the sparsity of named entity datasets and the uneven distribution of entity categories, traditional binary cross entropy loss functions require multiple rounds of training to decode entities. In this paper, we have modified the binary cross entropy loss function to make the proposed model faster decode the entities that need to be identified. The model performs well, according to the experimental findings. The approach suggested in this paper offers a fresh approach to the NER issue raised by electronic medical records, and it is anticipated to considerably boost the effectiveness and caliber of clinical medical research.
Privacy-preserving graph convolution network for federated item recommendation

In traditional recommender systems, we often build models based on a centralized storage of user data, which however will lead to user privacy concerns and risks. In this paper, we study an emerging and important recommendation problem called federated item recommendation (FIR), in which a recommendation model is built with decentralized data of user-item interactions in a privacy-aware manner, i.e., the personal behavior data of each user does not leave the owner. Recently, graph neural network (GNN) has been widely recognized as a state-of-the-art solution for item recommendation with implicit feedback since it is able to model the high-order connectivity between users and items. However, it is very challenging to exploit the high-order connectivity information in a decentralized user-item interaction graph without compromising user privacy. To address that, we propose a GNN-based federated recommendation framework, i.e., privacy-preserving graph convolution network (P-GCN), for the studied problem of FIR. Our P-GCN can leverage the high-order connectivity information like a centralized GCN model such as LightGCN, though it is built using a decentralized user-item graph. To achieve that, we design a novel privacy-preserving graph convolution approach based on secure aggregation and employ item-based user representation to compensate for the performance loss it causes due to the protection of user privacy. Moreover, we improve a group-wise concealing strategy for protecting user privacy. Empirical studies on three datasets show that our P-GCN can achieve similar or even better performance comparing with the non-federated (i.e., centralized) counterpart, and outperforms all the existing federated methods for the studied problem.
Stroke Prediction Using the Trust Evaluation with Data Leakage Avoiding

Abstract Early detection of the severe disease -- stroke is a key step toward effective treatment. Stroke disease data is imbalanced and normally contains the majority of negative cases (without stroke) and the minority of positive cases (stroke). Previous work has used SMOTE to deal with imbalanced data, but most researchers have implemented it for the entire dataset, which means the “answer” was silently “be told” and saved in the entire data, causing data leakage. Moreover, the previous work uses accuracy only as the metrics make the result less guaranteed. We propose a method using the SMOTE applied to the training set only and apply 13 machine learning classifiers for predicting stroke. We combine the AUC with accuracy as evaluation metrics in the stroke prediction task, which can elevate the confidence level of the assessment of results. The experiment shows that misused SMOTE and standardization can cause data leakage and the combined metrics can evaluate models with higher trustworthiness. We conclude that using our method can avoid data leakage and assess the model with higher trustworthiness.
Counterfactual explanations for misclassified images: How human and machine explanations differ

Counterfactual explanations have emerged as a popular solution for the eXplainable AI (XAI) problem of elucidating the predictions of black-box deep-learning systems because people easily understand them, they apply across different problem domains and seem to be legally compliant. Although over 100 counterfactual methods exist in the XAI literature, each claiming to generate plausible explanations akin to those preferred by people, few of these methods have actually been tested on users (∼7%). Even fewer studies adopt a user-centered perspective; for instance, asking people for their counterfactual explanations to determine their perspective on a "good explanation". This gap in the literature is addressed here using a novel methodology that (i) gathers human-generated counterfactual explanations for misclassified images, in two user studies and, then, (ii) compares these human-generated explanations to computationally-generated explanations for the same misclassifications. Results indicate that humans do not "minimally edit" images when generating counterfactual explanations. Instead, they make larger, "meaningful" edits that better approximate prototypes in the counterfactual class. An analysis based on "explanation goals" is proposed to account for this divergence between human and machine explanations. The implications of these proposals for future work are discussed.
EEP-3DQA: Efficient and Effective Projection-Based 3D Model Quality Assessment

Currently, great numbers of efforts have been put into improving the effectiveness of 3D model quality assessment (3DQA) methods. However, little attention has been paid to the computational costs and inference time, which is also important for practical applications. Unlike 2D media, 3D models are represented by more complicated and irregular digital formats, such as point cloud and mesh. Thus it is normally difficult to perform an efficient module to extract quality-aware features of 3D models. In this paper, we address this problem from the aspect of projection-based 3DQA and develop a no-reference (NR) Efficient and Effective Projection-based 3D Model Quality Assessment (EEP-3DQA) method. The input projection images of EEP-3DQA are randomly sampled from the six perpendicular viewpoints of the 3D model and are further spatially downsampled by the grid-mini patch sampling strategy. Further, the lightweight Swin-Transformer tiny is utilized as the backbone to extract the quality-aware features. Finally, the proposed EEP-3DQA and EEP-3DQA-t (tiny version) achieve the best performance than the existing state-of-the-art NR-3DQA methods and even outperforms most full-reference (FR) 3DQA methods on the point cloud and mesh quality assessment databases while consuming less inference time than the compared 3DQA methods.
Label-Semantic-Enhanced Online Hashing for Efficient Cross-modal Retrieval

Existing online cross-modal hashing methods often treat the semantic label categories independently to correlate the semantically similar data instances, which intrinsically ignore the potential dependency between the label categories and thus fail to capture the discriminative information in the hash code learning process. To alleviate this concern, we explore the inter-dependency between the label categories through their co-occurrence correlation from the label set, and present an efficient Label-Semantic-Enhanced Online Hashing (LSE-OH) method for various cross-modal retrieval task. To be specific, the proposed framework integrates the instance-wise similarity and label-category affinity to incrementally learn the discriminative hash codes for the current arriving data, while updating the hash functions at a streaming manner. Further, an iterative discrete optimization algorithm is derived to mine the inter-dependency between the label categories and discriminatively learn the hash codes without relaxation. Accordingly, the hash codes are adaptively learned online with the high discriminative capability and inter-dependency, while avoiding high computation complexity to process the streaming data. Experimental results show its outstanding performance in comparison with the-state-of-arts.
EANet: Towards Lightweight Human Pose Estimation With Effective Aggregation Network

Existing solutions to lightweight human pose estimation typically adopt a depthwise separable strategy, i.e., a normal 2D convolution is factorized into channel aggregation and spatial aggregation. However, this strategy cannot well capture multi-scale Effective Receptive Field (ERF), which is essential to dense prediction tasks like human pose estimation. To address this issue, we propose a novel lightweight network for human pose estimation, namely effective aggregation net (EANet). In EANet, we introduce two lightweight computational units: effective channel aggregating (ECA) and effective spatial aggregating (ESA), which are respectively responsible for channel-wise feature aggregation and pixel-wise feature aggregation. Unlike typical channel-wise aggregation using pointwise (1 × 1) convolution, the ECA aggregates few feature points that are estimated as effective ones. Moreover, the ESA is designed with re-parameterizing techniques, and it aggregates effective spatial feature points with multi-scale shared convolutions. Comprehensive experiments are conducted on three challenging datasets, i.e., COCO, Crowd-Pose, Wholebody-COCO. Our EANet demonstrates superior results on human pose estimation over previous lightweight methods, reaching a new state-of-the-art performance with a good trade-off. Our code and models are publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
MRSN: Multi-Relation Support Network for Video Action Detection

Action detection is a challenging video understanding task, requiring modeling spatio-temporal and interaction relations. Current methods usually model actor-actor and actor-context relations separately, ignoring their complementarity and mutual support. To solve this problem, we propose a novel network called Multi-Relation Support Network (MRSN). In MRSN, Actor-Context Relation Encoder (ACRE) and Actor-Actor Relation Encoder (AARE) model the actor-context and actor-actor relation separately. Then Relation Support Encoder (RSE) computes the supports between the two relations and performs relation-level interactions. Finally, Relation Consensus Module (RCM) enhances two relations with the long-term relations from the Long-term Relation Bank (LRB) and yields a consensus. Our experiments demonstrate that modeling relations separately and performing relation-level interactions can achieve and outperformer state-of-the-art results on two challenging video datasets: AVA and UCF101-24.
Is Really Correlation Information Represented Well in Self-Attention for Skeleton-based Action Recognition?

Transformer has shown significant advantages by various vision tasks. However, the lack of representation of correlation information about data properties makes it difficult to match the excellent results consistent with GCNs in skeleton-based action recognition. In this paper, we propose a Topology and Frames-guided Spatial-Temporal ConvFormer Network (TF-STCFormer), which is well suited for dynamically extracting topological and inter-frame uniqueness & co-occurrence information. Three essential components make up the proposed framework: (1) Grouped Physical-guided Spatial Transformer for focusing on learning essential spatial features and physical topology. (2) Global and Focal Temporal Transformer for promoting the relationship of different joints in consecutive frames and improving the representation of discriminative key-frames. (3) Grouped Dilation Temporal Convolution for connecting the intermediate output obtained by the previous transformers in the feature channels of different dilation. Experiments on four standard datasets (NTU RGB+D, NTU RGB+D 120, NW-UCLA, and UAV-Human) demonstrate that our approach prominently outperforms state-of-the-art methods on all benchmarks.
Dynamic Spatial-temporal Hypergraph Convolutional Network for Skeleton-based Action Recognition

Skeleton-based action recognition relies on the extraction of spatial-temporal topological information. Hypergraphs can establish prior unnatural dependencies for the skeleton. However, the existing methods only focus on the construction of spatial topology and ignore the time-point dependence. This paper proposes a dynamic spatial-temporal hypergraph convolutional network (DST-HCN) to capture spatial-temporal information for skeleton-based action recognition. DST-HCN introduces a time-point hypergraph (TPH) to learn relationships at time points. With multiple spatial static hypergraphs and dynamic TPH, our network can learn more complete spatial-temporal features. In addition, we use the high-order information fusion module (HIF) to fuse spatial-temporal information synchronously. Extensive experiments on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets show that our model achieves state-of-the-art, especially compared with hypergraph methods.
Accelerating Diffusion Sampling with Classifier-based Feature Distillation

Although diffusion model has shown great potential for generating higher quality images than GANs, slow sampling speed hinders its wide application in practice. Progressive distillation is thus proposed for fast sampling by progressively aligning output images of N-step teacher sampler with N/2-step student sampler. In this paper, we argue that this distillation-based accelerating method can be further improved, especially for few-step samplers, with our proposed Classifier-based Feature Distillation (CFD). Instead of aligning output images, we distill teacher’s sharpened feature distribution into the student with a dataset-independent classifier, making the student focus on those important features to improve performance. We also introduce a dataset-oriented loss to further optimize the model. Experiments on CIFAR-10 show the superiority of our method in achieving high quality and fast sampling. Code is available at https://github.com/zju-SWJ/RCFD.
General GAN-generated Image Detection by Data Augmentation in Fingerprint Domain

In this work, we investigate improving the generalizability of GAN-generated image detectors by performing data augmentation in the fingerprint domain. Specifically, we first separate the fingerprints and contents of the GAN-generated images using an autoencoder based GAN fingerprint extractor, followed by random perturbations of the fingerprints. Then the original fingerprints are substituted with the perturbed fingerprints and added to the original contents, to produce images that are visually invariant but with distinct fingerprints. The perturbed images can successfully imitate images generated by different GANs to improve the generalization of the detectors, which is demonstrated by the spectra visualization. To our knowledge, we are the first to conduct data augmentation in the fingerprint domain. Our work explores a novel prospect that is distinct from previous works on spatial and frequency domains augmentation. Extensive cross-GAN experiments demonstrate the effectiveness of our method compared to the state-of-the-art methods in detecting fake images generated by unknown GANs.