Reviewerly/exHarmony huggingface dataset:
TriAxial Low-Rank Transformer for Efficient Medical Image Segmentation

Transformer-CNN architectures have achieved state-of-the-art on 3D medical image segmentation due to their ability to capture both long-term dependencies and local information. However, directly using the existing transformers as encoders can be inefficient, particularly when dealing with high-resolution 3D medical images. This is due to the fact that self-attention computes pixel-to-pixel relationships, which is computationally expensive. Despite attempts to mitigate this through the use of local-window attention or axial-wise attention, these methods may result in the loss of interaction between certain local regions during the self-attention computation. Instead of using the sparsified attention, we aim to incorporate the relationships between all pixels while substantially reducing the computational demand. Inspired by the low-rank property of attention, we hypothesized that the pixel-to-pixel relationship can be approximated by the composition of the plane-to-plane relationship. We propose TriAxial Low-Rank Transformer Network (TALoRT-Net) for medical image segmentation. The core of this model lies in its attention module, which approximates pixel-to-pixel attention matrix using the low-rank representation of the product of plane-to-plane matrices and significantly reduces the computation complexity inherent in 3D self-attention. Moreover, we replaced the linear projection and vanilla Multi-Layer Perceptron (MLP) in Vision Transformer with a convolutional stem and depthwise convolution layer (DCL) to further reduce the number of model parameters. We evaluated the performance of the method on the public BTCV dataset, which significantly reduce the computational effort while maintaining uncompromised accuracy.
Matching-to-Detecting: Establishing Dense and Reliable Correspondences Between Images

We present a novel view for local image feature matching, which is inspired by the difference between existing methods. Detector-based methods detect predefined keypoints in local regions, so that the stability and reliability of established matches are ensured. In contrast, detector-free methods usually directly match dense features and refine the filtered results, which can help generate more matches. In order to combine their advantages, we propose a novel Matching-to-Detecting (M2D) process for feature matching, in which we first perform global reasoning for patch-level matching and subsequently identify discriminative matches within local areas to obtain pixel-level matches. At the patch-level, dense matching provides our pipeline with the ability to find plenty of matches even in low-texture areas, while at the pixel-level, our method can be viewed as detecting from a matching perspective, so that the established matches have higher stability and reliability and are remarkable in local regions. Experimental results demonstrate that our method outperforms state-of-the-art methods by a significant margin in terms of matching accuracy and the number of matches. Moreover, the computational complexity of our model is quite low, making it more suitable for real-world applications.
Sonic labyrinths: form and creative process in <i>like knotted strings</i> (2022) for string quartet

Abstractlike knotted strings for string quartet was commissioned in 2020 by the New York City-based Jack Quartet (through their Jack Studio programme), completed in 2022 and premiered in January 2023. In Jorge Luis Borges’s short story, The Garden of Forking Paths (2018 [1941]), the character Stephen Albert describes the fiction of the author Ts’ui Pen as works which explore all the possible narrative alternatives, in contrast with other authors whose fiction considers only a single outcome. Gary Saul Morson (1994:19), addressing a similar subject, argues that narrative offers both advantages and challenges to our comprehension of time. The metatextual nature of these themes - infinity (related here to narrative possibilities), memory and perception - gave me the impetus to compose like knotted strings, where they are reflected in such a way that musical material comments on the creative process itself.
Temporal-Channel Topology Enhanced Network for Skeleton-Based Action Recognition

Skeleton-based action recognition has become popular in recent years due to its efficiency and robustness. Most current methods adopt graph convolutional network (GCN) for topology modeling, but GCN-based methods are limited in long-distance correlation modeling and generalizability. In contrast, the potential of convolutional neural network (CNN) for topology modeling has not been fully explored. In this paper, we propose a novel CNN architecture, Temporal-Channel Topology Enhanced Network (TCTE-Net), to learn spatial and temporal topologies for skeleton-based action recognition. The TCTE-Net consists of two modules: the Temporal-Channel Focus module, which learns a temporal-channel focus matrix to identify the most important feature representations, and the Dynamic Channel Topology Attention module, which dynamically learns spatial topological features, and fuses them with an attention mechanism to model long-distance channel-wise topology. We conduct experiments on NTU RGB+D, NTU RGB+D 120, and FineGym datasets. TCTE-Net shows state-of-the-art performance compared to CNN-based methods and achieves superior performance compared to GCN-based methods. The code is available at https://github.com/aikuniverse/TCTE-Net .
Uncertainty-Aware Boundary Attention Network for Real-Time Semantic Segmentation

Remarkable progress has been made in real-time semantic segmentation by leveraging lightweight backbone networks and auxiliary low-level training tasks. Despite several techniques have been proposed to mitigate accuracy degradation resulting from model reduction, challenging regions often exhibit substantial uncertainty values in segmentation results. To tackle this issue, we propose an effective structure named Uncertainty-aware Boundary Attention Network(UBANet). Specifically, we model the segmentation uncertainty via prediction variance during training and involve it as a regularization item into optimization objective to improve segmentation performance. Moreover, we employ uncertainty maps to investigate the role of low-level supervision in segmentation task. And we reveal that directly fusing high- and low-level features leads to the overshadowing of large-scale low-level features by the encompassing local contexts, thus hindering the synergy between the segmentation task and low-level tasks. To address this issue, we design a Low-level Guided Feature Fusion Module that avoids the direct fusion of high- and low-level features and instead employs low-level features as guidance for the fusion of multi-scale contexts. Extensive experiments demonstrate the efficiency and effectiveness of our proposed method by achieving the state-of-the-art latency-accuracy trade-off on Cityscapes and CamVid benchmark.
Deep Consistency Preserving Network for Unsupervised Cross-Modal Hashing

Given the proliferation of multimodal data in search engines and social networks, unsupervised cross-modal hashing has gained traction for its low storage consumption and fast retrieval speed. Despite the great success achieved, unsupervised cross-modal hashing still suffers from lacking reliable similarity supervision and struggles with reducing information loss caused by quantization. In this paper, we propose a novel deep consistency preserving network (DCPN) for unsupervised cross-modal hashing, which sufficiently utilizes the semantic information in different modalities. Specifically, we gain consistent features to fully exploit the co-occurrence information and alleviate the heterogeneity between different modalities. Then, a fusion similarity matrix construction method is proposed to capture the semantic relationship between instances. Finally, a fusion hash code reconstruction strategy is designed to fit the gap between different modalities and reduce the quantization error. Experimental results demonstrate the effectiveness of the proposed DCPN on unsupervised cross-modal retrieval tasks.
Unsupervised Prototype Adapter for Vision-Language Models

Recently, large-scale pre-trained vision-language models (e.g. CLIP and ALIGN) have demonstrated remarkable effectiveness in acquiring transferable visual representations. To leverage the valuable knowledge encoded within these models for downstream tasks, several fine-tuning approaches, including prompt tuning methods and adapter-based methods, have been developed to adapt vision-language models effectively with supervision. However, these methods rely on the availability of annotated samples, which can be labor-intensive and time-consuming to acquire, thus limiting scalability. To address this issue, in this work, we design an unsupervised fine-tuning approach for vision-language models called Unsupervised Prototype Adapter (UP-Adapter). Specifically, for the unannotated target datasets, we leverage the text-image aligning capability of CLIP to automatically select the most confident samples for each class. Utilizing these selected samples, we generate class prototypes, which serve as the initialization for the learnable prototype model. After fine-tuning, the prototype model prediction is combined with the original CLIP's prediction by a residual connection to perform downstream recognition tasks. Our extensive experimental results on image recognition and domain generalization show that the proposed unsupervised method outperforms 8-shot CoOp, 8-shot Tip-Adapter, and also the state-of-the-art UPL method by large margins.
Enhancing Recommender System with Multi-modal Knowledge Graph

Recommender systems have shown great potential to solve the problem of information overload and improve user experience in various online applications. To address the data sparsity and cold start problems in these systems, researchers have proposed leveraging knowledge graphs (KGs) as auxiliary information for recommendations, which contain valuable external knowledge. However, most of these works have neglected the diversity of data types, such as texts and images, in multi-modal knowledge graphs (MMKGs). In this paper, we propose the Multi-modal Knowledge Graph Attention Network to enhance recommender systems by leveraging multi-modal knowledge. Specifically, We propose a multi-modal knowledge graph attention mechanism to facilitate knowledge propagation over MMKGs. The resulting knowledge-enhanced entity embedding representations are utilized for recommendation. Extensive experiments conducted on MovieLens datasets demonstrate the rationality and effectiveness of our model.
Node and edge dual-masked self-supervised graph representation

Abstract Self-supervised graph representation learning has been widely used in many intelligent applications since labeled information can hardly be found in these data environments. Currently, masking and reconstruction-based (MR-based) methods lead the state-of-the-art records in the self-supervised graph representation field. However, existing MR-based methods did not fully consider both the deep-level node and structure information which might decrease the final performance of the graph representation. To this end, this paper proposes a node and edge dual-masked self-supervised graph representation model to consider both node and structure information. First, a dual masking model is proposed to perform node masking and edge masking on the original graph at the same time to generate two masking graphs. Second, a graph encoder is designed to encode the two generated masking graphs. Then, two reconstruction decoders are designed to reconstruct the nodes and edges according to the masking graphs. At last, the reconstructed nodes and edges are compared with the original nodes and edges to calculate the loss values without using the labeled information. The proposed method is validated on a total of 14 datasets for graph node classification tasks and graph classification tasks. The experimental results show that the method is effective in self-supervised graph representation. The code is available at: https://github.com/TangPeng0627/Node-and-Edge-Dual-Mask .
Sem-Avatar: Semantic Controlled Neural Field for High-Fidelity Audio Driven Avatar

In this paper, we tackle the audio-driven avatar challenge by fitting a semantic controlled neural field to a talking-head video. While existing methods struggle with realism and head-torso inconsistency, our novel end-to-end framework, semantic controlled neural field (Sem-Avatar) sucessfully overcomes the above problems, delievering high-fidelity avatar. Specifically, we devise a one-stage audio-driven forward deformation approach to ensure head-torso alignment. We further propose to use semantic mask as a control signal for eye opening, lifting the naturalness of the avatar to another level. We train our framework via comparing the rendered avatar to the original video. We further append a semantic loss which leverages human face prior to stabilize training. Extensive experiments on public datasets demonstrate Sem-Avatar’s superior rendering quality and lip synchronization, establishing a new state-of-the-art for audio-driven avatars.
Advancing medical data classification through federated learning and blockchain incentive mechanism: implications for modern software systems and applications

Abstract The key issue of medical data is patient information sensitivity and dataset finiteness, which need to guarantee high-efficient training. Besides, the current convolutional neural network has a low image classification and poor robustness concerning antagonistic samples. A lack of scalability in healthcare federated learning and incentive mechanism hinders the attraction of ample high-quality datasets. This paper proposes a Federated Learning Incentive Mechanism for Medical Data Classification (FedIn-MC). It realizes a collaborative model training of multi-party medical institutions through the combination of federated learning and blockchain. There is a marked improvement to the model’s robustness through a combination of the distance loss function and the prototype loss regulation. In addition, this incentive mechanism of blockchain in the project is applied to calculate client contribution values and encourage healthcare institutions to active training model participation. Simulation results verify an accomplishment of a multi-party training. With regard to image classifications, this framework also has a higher classification accuracy and stronger robustness concerning invisible class samples.
Cavity‐Assisted Atomic External Momenta State Teleportation

Abstract The quantized momentum eigenstates of neutral atoms offer high resistance to decoherence and thus suggests a viable alternative for quantum informatics tasks including teleportation. In this work, a scheme is suggested to teleport an unknown superposition of two distinct external momentum states of a neutral atom onto the similar momentum states of a distant atom at the receiving end through the off‐resonant atomic Bragg diffraction (ABD). In order to teleport this unknown state from the sender, that is, Alice, to the receiver, that is, Bob, a pre‐existing entangled link formed through a high‐Q cavity at Alice's end and momentum states of a neutral atom at the Bob's end are utilized. Further, while citing the realistic experimental parameters, it is demonstrated that the proposal can easily be implemented experimentally under the prevailing cavity‐QED atom‐field research scenario. The proposal is also generalized to cover the teleportation of the multipartite entangled momenta states and the case of multipartite GHZ entangled states is elucidated in details. However, the procedure is generic and can be extended to cover the teleportation schematics for any arbitrary atomic momenta states.
Scalable Quantum Spin Networks from Unitary Construction

Abstract Spin network (SN) systems can be used to achieve quantum state transfer with high fidelity and to generate entanglement. A new approach to design spin‐chain‐based spin network systems, for short‐range quantum information processing and phase‐sensing, has been proposed recently in Advanced Quantum Technologies . In this paper, the scalability of such systems is investigated, by designing larger SN systems that can be used for longer‐range quantum information tasks, such as connecting together quantum processors. Furthermore, more complex SN designs, which can produce different types of entangled states, are presented. Simulations of disorder effects show that even such larger SN systems are robust against realistic levels of disorder.
Towards a Framework for a Nation-Wide Implementation of Augmented, Virtual and Mixed Reality in K-12 Technical and Vocational Education

Abstract As augmented, virtual and mixed reality have become more user-friendly and affordable, these technologies gained increasing interest from education. Teachers all over the world are triggered by the perceived benefits and start experimenting. However, teachers encounter obstacles to pursue effective implementation. This paper describes how these obstacles are being tackled in Flanders (Belgium) via a large-scale, nation-wide framework for the implementation of augmented, virtual and mixed reality in K-12 technical and vocational education. This framework was designed, adopting an Educational Design Research approach, and consists of five interrelated pillars: hardware, software, professional development of teachers, practice-oriented research, and coordination. The proposed framework provides guidelines, both for researchers and education policy makers.
Physical adversarial attack in artificial intelligence of things

Abstract With the continuous development of wireless communication and artificial intelligence technology, Internet of Things (IoT) technology has made great progress. Deep learning methods are currently used in IoT technology, but deep neural networks (DNNs) are notoriously susceptible to adversarial examples, and subtle pixel changes to images can result in incorrect recognition results from DNNs. In the real‐world application, the patches generated by the recent physical attack methods are larger or less realistic and easily detectable. To address this problem, a Generative Adversarial Network based on Visual attention model and Style transfer network (GAN‐VS) is proposed, which reduces the patch area and makes the patch more natural and less noticeable. A visual attention model combined with generative adversarial network is introduced to detect the critical regions of image recognition, and only generate patches within the critical regions to reduce patch area and improve attack efficiency. For any type of seed patch, an adversarial patch can be generated with a high degree of stylistic and content similarity to the attacked image by generative adversarial network and style transfer network. Experimental evaluation shows that the proposed GAN‐VS has good camouflage and outperforms state‐of‐the‐art adversarial patch attack methods.
Multi-scale Dilated Attention Graph Convolutional Network for Skeleton-Based Action Recognition

Due to the small size, anti-interference and strong robustness of skeletal data, research on human skeleton-based action recognition has become a mainstream. However, due to the incomplete utilization of semantic information and insufficient time modeling, most methods may not be able to fully explore the connections between non-adjacent joints in the spatial or temporal dimensions. Therefore, we propose a Multi-scale Dilated Attention Graph Convolutional Network for Skeleton-Based Action Recognition (MDKA-GCN) to solve the above problems. In the spatial configuration, we explicitly introduce the channel graph composed of high-level semantics (joint type and frame index) of joints into the network to enhance the representation ability of spatiotemporal features. MDKA-GCN uses joint-level, velocity-level and bone-level graphs to more deeply mine the hidden features of human skeletons. In the time configuration, two lightweight multi-scale strategies are proposed, which can be more robust to time changes. Extensive experiments on NTU-RGB+D 60 datasets and NTU-RGB+D 120 datasets show that MDKA-GCN has reached an advanced level, and surpasses the performance of most lightweight SOTA methods.
SwinFusion: Channel Query-Response Based Feature Fusion for Monocular Depth Estimation

Given a single RGB image, Monocular Depth Estimation (MDE) predicts the depth of each pixel by utilizing both global and local information. Most MDE networks use simple skip connections, each skip connection only contain single-scale feature information. In this paper, we propose a multi-scale feature fusion module called Channel Query-Response Fusion (CQRF), which uses a channel query-response mechanism to filter out harmful or invalid channel information and enables soft fusion on the channels of multi-scale features. It can be inserted as an independent module into other MDE network architectures and enhance their performance. After passing through the CQRF module, the skip connections will contain valid channel information from all scales. Based on CQRF, we design a multi-scale feature fusion network SwinFusion, which achieves advanced results on public dataset. Additionally, we analyzed the effects of the number of utilized multi-scale features and different layer channels on the performance of MDE task.
Efficient Adversarial Training with Membership Inference Resistance

Deep cross-media computing faces adversarial example attacks, adversarial training is an effective approach to enhance the robustness of machine learning models via adding adversarial examples into the training phase. However, existing adversarial training methods increase the advantage of membership inference attacks, which aim to determine from the model whether an example is in the training dataset. In this paper, we propose an adversarial training framework that guarantees both robustness and membership privacy by introducing a tailor-made example, called reverse-symmetry example. Moreover, our framework reduces the number of required adversarial examples compared with existing adversarial training methods. We implement the framework based on three adversarial training methods on FMNIST and CIFAR10. The experimental results show that our framework outperforms the original adversarial training with respect to the overall performance of accuracy, robustness, privacy, and runtime.
Modality Interference Decoupling and Representation Alignment for Caricature-Visual Face Recognition

Cross-modality face recognition aims to match facial images across different modalities. This task becomes very challenging when one of the modalities is the facial caricature, which enhances instinctive facial features through extreme distortions and exaggerations with diverse styles by artists. In this paper, we develop a novel modality interference decoupling and representation alignment (MIR) method for visual-caricature face recognition. Our MIR method consists of a backbone network, an identity-interference orthogonal decoupling (IIOD) module, and a modality feature alignment (MFA) module. The IIOD module adopts a three-branch structure to decouple the deep semantic features extracted by the backbone network into identity features and modality features. In IIOD, we design an identity subspace alignment (ISA) module to align the identity features from different branches. Moreover, we design the MFA module to perform feature alignment between the modality feature from the IIOD module and that from the pre-trained modality interference information encoder (MIE) via adversarial learning, extracting the modality-specific information. Based on the above designs, we can effectively alleviate the interference of modality differences and style differences, improving the final performance. Extensive experimental results on multiple datasets show that our proposed method outperforms several state-of-the-art cross-modality face recognition methods.