Reviewerly/exHarmony huggingface dataset:
Attention and Relative Distance Alignment for Low-Resolution Facial Expression Recognition

In real-world scenarios, facial images obtained by many devices often exhibit low resolution. However, the performance significantly degrades when we apply the existing methods in low-resolution facial expression recognition. Therefore, addressing the problem of low-resolution images in facial expression recognition becomes an important undertaking. Previous attempts to tackle this problem have been limited. For this, we propose a novel Attention and Relative Distance Alignment (ARDA) method by integrating knowledge distillation in low-resolution facial expression recognition. Specifically, the Attention Alignment module guides the student model to focus on the most crucial region of the facial image by enabling the low-resolution student model to learn the attention map of the high-resolution teacher model. The Relative Distance Alignment module utilizes the relative distance between facial image features to transfer differences between different low-resolution facial images from the teacher model to the student model, helping the student model better grasp the differences between expressions. Extensive experiments have shown that the ARDA method effectively transfers knowledge from high-resolution teacher model to low-resolution student model, achieving state-of-the-art performance in synthetic low-resolution facial expression recognition datasets.
Multi-behavior Enhanced Graph Neural Networks for Social Recommendation

Social recommendation has gained more and more attention by utilizing the social relationships among users, alleviating the data sparsity problem in collaborative filtering. Most existing social recommendation approaches treat the preference propagation process coarse-grained, ignoring the different diffusion patterns targeting corresponding interaction behaviors. However, this may be inappropriate because of the interplay between multi-behavior and social relations. Therefore, in this paper, we propose a novel framework, MB-Soc, for Multi-Behavior Enhanced Social Recommender, to model the mutual effect between users’ multiple behaviors and social connections. In MB-Soc, we first devise a single behavior-based social diffusion module to depict behavioral trust propagation. Moreover, to support behavior integration, we propose an intent embedding to ensure behavior independency. In addition, we design a Self-Supervised Learning-based behavior integration module to capture the correlations among multiple behaviors. Extensive experiments conducted on two real-world datasets demonstrate the effectiveness of our model.
“Strategic framing” as a modern marketing tool for book publishing management

Aim. To study the book market as one of the most specific branch markets where scaling is impossible and the only way to compete is to escape from it, and on the basis of the analysis to give an assessment of the competitive environment of independent bookstores in St. Petersburg. Objectives. To define the methodology of competitive environment analysis; to characterize the market of independent bookstores in St. Petersburg; to analyze the competitive environment of the most significant independent bookstores. Methods. The authors used general scientific methods: deduction, analysis and synthesis, logical analysis, methods of marketing research. Results. The strategic framing provides an opportunity to most fully assess the potential not only of a particular enterprise, but also of the industry as a whole. It allows you to assess the similarity and difference of your strategy in comparison with the strategies of competitors. All competitive niches are used by stores to a greater or lesser extent, which means that a new point of sale, in case of entering such a market, should offer innovation of value, and at least at the level of overlap with the practices of other industries. Conclusions. We have analyzed a segment of the book market - the market of independent bookstores. The stores were divided into competing groups, and their competitive advantages were evaluated using the so-called strategic framework method, which allowed us to determine that the modern market segment has exhausted the potential of the main value innovations proposed at the initial formation. On the one hand, this means increased competition and the need for old players to offer better service, on the other hand - the need for new players to create new values for the consumer.
Fuse Tune: Hierarchical Decoder Towards Efficient Transfer Learning

This paper shows that integrating intermediate features brings superior efficiency to transfer learning without hindering performance. The implementation of our proposed Fuse Tune is flexible and simple: we preserve and concatenate features in the pre-trained backbone during forward propagation. A hierarchical feature decoder with shallow attentions is then designed to further dig the rich relations across features. Since the deep buried features are entirely exposed, back propagation in the cumbersome backbone is no longer required, which greatly improves the training efficiency. We make the observation that Fuse Tune performs especially well in self-supervised and label-scarce scenarios, proving its adaptive and robust representation transfer ability. Experiments demonstrate that Fuse Tune could achieve $$184\%$$ temporal efficiency and $$169\%$$ memory efficiency lead compared to Prompt Tune in ImageNet. Our code is open sourced in https://github.com/JeavanCode/FuseTune .
Mining Temporal Inconsistency with 3D Face Model for Deepfake Video Detection

Recently, the abuse of face swapping technique including Deepfakes has garnered widespread attention. Facial manipulation techniques arise serious problem, including information confusion and erosion of trust. Previous methods for detecting deepfakes have primarily focused on identifying forged traces in independent frames or from two-dimensional face. As a result, the challenge of detecting forged traces from video sequence and higher dimensions has emerged as a critical research area. In this paper, we approach face swapping detection from a novel perspective by converting face frames into three-dimensional space. We argue that the Euclidean distance between landmarks in 2D faces and 3D facial reprojections, along with the shape and texture feature of the 3D face model, serve as clues to discern authenticity. Our framework incorporates a Recurrent Neural Network (RNN) to effectively exploit temporal features from videos. Extensive experiments conducted on several datasets demonstrate the effectiveness of our method. Particularly in cross-manipulation experiments, our approach outperforms state-of-the-art competitors, highlighting its potential as a robust solution for detecting deepfakes.
Multi-depth Fusion Transformer and Batch Piecewise Loss for Visual Sentiment Analysis

Visual sentiment analysis is an important research direction in the field of affective computing, which is of great significance in user behavior prediction, visual scene construction, etc. In order to address the issue of low efficiency in high-level semantic perception of CNNs, in this paper, the Transformer architecture is introduced into the task of visual sentiment analysis, proposing the Multi-depth Fusion Transformer (MFT) model that allows nesting different architectures. Additionally, addressing the problem of imbalanced data samples and varying learning difficulties in visual sentiment analysis datasets, this paper proposes a batch piecewise loss suitable for visual sentiment binary classification tasks. Experimental results on multiple datasets demonstrate that the proposed method outperforms existing approaches. Visualization experiments comparing CNN and Transformer-based approaches demonstrate the effectiveness of the proposed method in semantic perception.
DARN: Crowd Counting Network Guided by Double Attention Refinement

Although great progress has been made in crowd counting, accurate estimation of crowd numbers in high-density areas and full mitigation of the interference of background noise remain challenging. To address these issues, we propose a method called Double Attention Refinement Guided Counting Network (DARN). DARN introduces an attention-guided feature aggregation module that dynamically fuses features extracted from the Transformer backbone. By adaptively fusing features at different scales, this module can estimate the crowd for high-density areas by restoring the lost fine-grained information. Additionally, we propose a segmentation attention-guided refinement method with multiple stages. In this refinement process, crowd background noise is filtered by introducing segmentation attention maps as masks, resulting in a significant refinement of the foreground features. The introduction of multiple stages can further refine the features by utilizing fine-grained and global information. Extensive experiments were conducted on four challenging crowd counting datasets: ShanghaiTech A, UCF-QNRF, JHU-CROWD++, and NWPU-Crowd. The experimental results validate the effectiveness of the proposed method.
Exploring Frequency Attention Learning and Contrastive Learning for Face Forgery Detection

Face forgery detection has become a critical security concern due to advances in manipulation techniques. Most methods look for forged clues from the spatial or vanilla frequency domain, leading to serious over-fitting. In this paper, we propose a Frequency Attention Module (FAM) that enhances model generalizability in face forgery detection. We theoretically demonstrate the feasibility of frequency attention learning, which allows the network to automatically refine subtle but discriminative forged features and suppress irrelevant components in the frequency domain without complex manual partitions. Besides, considering that commonly-used cross-entropy loss neglects the intra-class compactness, we design the DeepFake Contrastive Loss (DFCL) to decrease intra-class variances for real faces and enlarge inter-class differences in the feature space. Extensive experiments show that our method significantly outperforms SoTA methods on widely-used benchmarks.
Self-supervised Cascade Training for Monocular Endoscopic Dense Depth Recovery

Dense depth prediction for 3-D reconstruction of monocular endoscopic images is an essential way to expand the surgical field and augment the perception of surgeons in robotic endoscopic surgery. However, it is generally challenging to precisely estimate the monocular dense depth and reconstruct such a field due to complex surgical fields with a limited field of view, illumination variations, and weak texture information. This work proposes a new framework of self-supervised learning with a two-stage cascade training strategy for dense depth recovery of monocular endoscopic images. While the first stage is to train an initial deep-learning model through sparse depth consistency supervision, the second stage introduces photometric consistency supervision to further train and refine the initial model for improving its capability. Our framework was evaluated on patient data of monocular endoscopic images acquired from colonoscopic procedures, with the experimental results demonstrating that our self-supervised learning model with cascade training provides a promising strategy outperforming other models. On the one hand, both visual quality and quantitative assessment of our method are better than current monocular dense depth estimation approaches. On the other hand, our method relies less on sparse depth data for supervision than other self-supervised methods.
VCD: Visual Causality Discovery for Cross-Modal Question Reasoning

Existing visual question reasoning methods usually fail to explicitly discover the inherent causal mechanism and ignore jointly modeling cross-modal event temporality and causality. In this paper, we propose a visual question reasoning framework named Cross-Modal Question Reasoning (CMQR), to discover temporal causal structure and mitigate visual spurious correlation by causal intervention. To explicitly discover visual causal structure, the Visual Causality Discovery (VCD) architecture is proposed to find question-critical scene temporally and disentangle the visual spurious correlations by attention-based front-door causal intervention module named Local-Global Causal Attention Module (LGCAM). To align the fine-grained interactions between linguistic semantics and spatial-temporal representations, we build an Interactive Visual-Linguistic Transformer (IVLT) that builds the multi-modal co-occurrence interactions between visual and linguistic content. Extensive experiments on four datasets demonstrate the superiority of CMQR for discovering visual causal structures and achieving robust question reasoning. The supplementary file can be referred to https://github.com/YangLiu9208/VCD/blob/main/0793_supp.pdf .
CTHPose: An Efficient and Effective CNN-Transformer Hybrid Network for Human Pose Estimation

Recently, CNN-Transformer hybrid network has been proposed to resolve either the heavy computational burden of CNN or the difficulty encountered during training the Transformer-based networks. In this work, we design an efficient and effective CNN-Transformer hybrid network for human pose estimation, namely CTHPose. Specifically, Polarized CNN Module is employed to extract the feature with plentiful visual semantic clues, which is beneficial for the convergence of the subsequent Transformer encoders. Pyramid Transformer Module is utilized to build the long-term relationship between human body parts with lightweight structure and less computational complexity. To establish long-term relationship, large field of view is necessary in Transformer, which leads to a large computational workload. Hence, instead of the entire feature map, we introduced a reorganized small sliding window to provide the required large field of view. Finally, Heatmap Generator is designed to reconstruct the 2D heatmaps from the 1D keypoint representation, which balances parameters and FLOPs while obtaining accurate prediction. According to quantitative comparison experiments with CNN estimators, CTHPose significantly reduces the number of network parameters and GFLOPs, while also providing better detection accuracy. Compared with mainstream pure Transformer networks and state-of-the-art CNN-Transformer hybrid networks, this network also has competitive performance, and is more robust to the clothing pattern interference and overlapping limbs from the visual perspective.
Teacher-Student Cross-Domain Object Detection Model Combining Style Transfer and Adversarial Learning

Cross-domain object detection is challenging because object detection models are significantly susceptible to domain style. As a popular semi-supervised learning method, the teacher-student framework (pseudo labels from the teacher model supervise the student model) achieves significant accuracy gains in cross-domain object detection. However, it suffers from the domain shift and prone to generate low-quality pseudo labels, which limits the performance. To mitigate this problem, we propose a teacher-student framework that utilizes style transfer method, augmentation strategies, and adversarial learning to address domain shift. Specifically, we design a Fourier style transfer method to reduce the gap between source and target domains without altering the semantic information of the objects. Furthermore, we improve the data augmentation strategy, by weakly augmenting the images from the target domain, to avoid the teacher model biased to the source domain. Finally, we employ feature-level adversarial training in the student model which is trained based on images from all domains, allowing features derived from all domains to share similar distributions. This process ensures that the student model produces domain-invariant features. Our approach achieves state-of-the-art performance in several benchmark tests. For example, it achieved 51.6% and 49.9% mAP on Foggy Cityscapes and Clipart1K, respectively.
Camouflaged Object Detection via Global-Edge Context and Mixed-Scale Refinement

Camouflage object detection (COD), trying to segment objects that blend perfectly with the surrounding environment, is challenging and complex in real-world scenarios. However, existing deep learning methods often fail to accurately identify the boundary detail branches and complete structure of camouflaged objects. To address these challenges, we propose a novel Global-edge Context and Mixed-scale Refinement Network (GCMRNet) to handle the challenging COD task. Specifically, we propose a Global-edge Context Module (GCM), to effectively obtain long-distance context dependencies, which provides rich global context information. In addition, we also propose the Hierarchical Mixed-scale Refinement Module (HMRM) to conduct information interaction and feature refinement between channels, which aggregates the rich multi-level features for accurate COD. Extensive experimental results on three challenging benchmark datasets demonstrate that our proposed method outperforms 21 state-of-art methods under widely used evaluation metrics.
Binarizing Super-Resolution Neural Network Without Batch Normalization

In this paper, our objective is to propose a model binarization method aimed at addressing the challenges posed by over-parameterized super-resolution (SR) models. Our analysis reveals that binary SR models experience significant performance degradation, primarily attributed to their sensitivity towards weight/activation distributions, particularly when devoid of Batch Normalization (BN) layers. Consequently, we undertake the following endeavors in this study: First, we conduct a comprehensive analysis to examine the impact of BN layers on SR models based on Binary Neural Networks (BNNs). Second, we propose an asymmetric binarizer that can be reparameterized to adaptively adjust the transition point for activation binarization. Third, we introduce a progressive gradient estimator that modifies weight smoothness and controls weight flipping to stabilize the training procedure in the absence of BN layers. Through extensive experiments, we demonstrate that our proposed method exhibits significant performance improvements. For instance, when binarizing EDSR and scaling up input images by a factor of $$\times 4$$ , our approach achieves a PSNR decrease of less than 0.4dB on the Urban100 benchmark.
Multimodal Topic and Sentiment Recognition for Chinese Data Based on Pre-trained Encoders

With the rapid development of mobile internet technology, massive amounts of multimodal data have emerged from major online platforms. However, defining multimodal topic categories is a more subjective task, and the lack of a public Chinese dataset hinders the progress of the task. And fine-grained multimodal sentiment classification is extremely challenging, especially for aspect-level classification. In this paper, we build a Chinese dataset based on Weibo for multimodal topic classification and propose a pre-trained encoder-based model for topic classification and fine-grained sentiment classification. The experimental results on Multi-ZOL and our proposed dataset show that the proposed model outperforms the benchmark model on the multi-modal topic classification task, achieving an accuracy of 0.99. On the fine-grained multimodal sentiment classification task, the proposed model achieves an accuracy of 0.5860 and a macro-F1 of 0.5791, which is competitive with SOTA.
Half Aggregation Transformer for Exposure Correction

Photos taken under poor illumination conditions often suffer from unsatisfactory visual effects. Recently, Transformer, avoiding the shortcomings of CNN models, has shown impressive performance on various computer vision tasks. However, directly leveraging Transformer for exposure correction is challenging. On the one hand, the global Transformer has an excessive computational burden and fails to preserve local feature details. On the other hand, the local Transformer fails to extract spatially varying light distributions. Both global illumination recovery and local detail enhancement are indispensable for exposure correction. In this paper, we propose a novel Half Aggregation Transformer (HAT) architecture for exposure correction with a key design of Half Aggregation Multi-head Self-Attention (HA-MSA). Specifically, our HA-MSA establishes inter- and intra-window token interactions via window aggregation and window splitting strategies to jointly capture both global dependencies and local contexts in a complementary manner. In addition, we customize an illumination guidance mechanism to explore illumination cues and boost the robustness of the network to handle complex illumination. Extensive experiments demonstrate that our method outperforms the state-of-the-art exposure correction methods qualitatively and quantitatively with cheaper computational costs.
A Video Face Recognition Leveraging Temporal Information Based on Vision Transformer

Video face recognition (VFR) has gained significant attention as a promising field combining computer vision and artificial intelligence, revolutionizing identity authentication and verification. Unlike traditional image-based methods, VFR leverages the temporal dimension of video footage to extract comprehensive and accurate facial information. However, VFR heavily relies on robust computing power and advanced noise processing capabilities to ensure optimal recognition performance. This paper introduces a novel length-adaptive VFR framework based on a recurrent-mechanism-driven Vision Transformer, termed TempoViT. TempoViT efficiently captures spatial and temporal information from face videos, enabling accurate and reliable face recognition while mitigating the high GPU memory requirements associated with video processing. By leveraging the reuse of hidden states from previous frames, the framework establishes recurring links between frames, allowing the modeling of long-term dependencies. Experimental results validate the effectiveness of TempoViT, demonstrating its state-of-the-art performance in video face recognition tasks on benchmark datasets including iQIYI-ViD, YTF, IJB-C, and Honda/UCSD.
Emotion Recognition via 3D Skeleton Based Gait Analysis Using Multi-thread Attention Graph Convolutional Networks

Human gait is a manner of individual’s walking that observers are able to learn useful information through daily walking activities. Recently, gait skeletons-based emotion recognition has attracted much attention, while many methods have been proposed gradually. Skeleton-based representations offer several advantages for recognition tasks. In particular, such representation is extremely lightweight and could be directly extracted from video data using off-the-shelf algorithms. Moreover, skeleton data is not tied to any specific cultural or ethnic context, it hence become increasingly popular in recent years for cross-cultural studies and other related applications. To effectively process this type of data, many researchers have turned to Graph Convolutional Networks (GCNs) to leverage the topological structure of the data, which improves performance by modeling the relationships between different joints and body parts as a graph. This allows GCNs to capture complex spatial and temporal patterns. In this work, we have constructed an efficient multi-stream GCN framework for emotion recognition task. We have identified the complementary effect among streams using a multi-thread attention method (MTA), which is able to improve the emotion recognition performance. In addition, the proposed MTA graph convolution layer is able to extract effective features from the topology of the graph to further improve recognition performance. The proposed method outperforms state-of-art methods on challenging benchmark dataset.