Reviewerly/exHarmony huggingface dataset:
ConCAP: Contrastive Context-Aware Prompt for Resource-hungry Action Recognition

Existing large-scale image-language pre-trained models, e.g., CLIP [1], have revealed strong spatial recognition capability on various vision tasks. However, they achieve inferior performance in action recognition due to lack of temporal reasoning ability. Moreover, fully tuning large models require expensive computational infrastructures, and state-of-the-art video models yield slow inference speed due to the high frame sampling rate. The above drawbacks make existing video action recognition works impractical to be applied in resource-hungry scenarios, which is common in the real world. In this work, we propose Contrastive Context-Aware Prompt (ConCAP) for resource-hungry action recognition. Specifically, we develop a lightweight PromptFormer to learn the spatio-temporal representations stacking on top of frozen frame-wise visual backbones, where learnable prompt tokens are plugged between frame tokens during self-attention. These prompt tokens are expected to auto-complete the contextual spatiotemporal information between frames and therefore enhance the model’s representation capability. To achieve this goal, we align the prompt-enhanced representation with both category-level textual representations and video representations from densely sampled frames. Extensive experiments on four video benchmarks show that we achieve state-of-the-art or competitive performance compared to existing methods with far fewer trainable parameters and faster inference speed with limited frames, demonstrating the superiority of ConCAP in resource-hungry scenarios.
SDGFormer: An Efficient Convolution Network Structurally Similar to Transformer

Deep Neural Networks (DNN) have achieved extraordinary success in many visual recognition tasks. Visual Transformer (ViT), which is derived from Natural Language Processing (NLP), has achieved state-of-the-art (SOTA) results on many tasks due to its capability of capturing long-range dependencies in visual data. However, Existing ViT models are challenging to deploy on devices due to their massive computational consumption, huge memory overhead, and reliance on large datasets. In this work, we address these issues by replacing some computationally expensive and memory-intensive modules in ViT with standard Convolutional Neural Network (CNN) modules. Firstly, we propose an efficient Self-Attention module called SDG-Attention (SDGA) with linear space and time complexity, and an economical FeedForward Network (FFN) composed of group convolution and shuffle channel (SFFN). Then, we develop a lightweight CNN model with SDGA and SFFN, SDGFormer, which embraces several priors of ViT and is LayerNorm-Free. We evaluate SDGFormer on ImageNet-1K and Mini-ImageNet, and the SDGFormer-S achieves a comparable top-1 accuracy of 77.6% on ImageNet-1K with 9.1M parameters and 1.6 GFlops regimes. Moreover, our SDGFormer-T achieves SOTA performance on Mini-ImageNet with 83.3% accuracy, demonstrating good generalization on small datasets without extra data.
A Retriever-Reader Framework with Visual Entity Linking for Knowledge-Based Visual Question Answering

In this paper, we propose a Retriever-Reader framework with Visual Entity Linking (RR-VEL) for knowledge-based visual question answering. Given images and original questions, the visual entity linking (VEL) module extracts key entities in images to replace the question referents for semantic disambiguation, achieving entity-oriented queries with explicit entities. Furthermore, the Retriever encodes the queries and knowledge items by Bert with a feed-forward layer, and obtains a set of knowledge candidates. The Reader encodes the questions with image captions and knowledge candidates in two branches, which avoids their interference during self-attentive encoding. Finally, the decoder of Reader fuses the encoded features to generate answers. Extensive experiments conducted on the two public datasets show that our method significantly outperforms the existing baselines.
SWATM: Contribution-Aware Adaptive Federated Learning Framework Based on Augmented Shapley Values

The vanilla federated learning (FedAvg) takes a weighted aggregation of uploaded models based on the amount of data from each participant. By addressing varying data quality and heterogeneous data distributions across participants, the existing work calculates the contribution of participants based on Shapley Value (SV) and attempt to guide model aggregation under SV. Our empirical study found that vanilla SV exhibits low sensitivity and instability issues. To address these issues, we propose a Shapley Value Weighted FL aggregation method based on Adaptive Temperature augmentation and Momentum update (SWATM). The adaptive temperature moderately augments the sensitivity of SV according to aggregation performance, while the momentum update slows down the fluctuation and improves stability. Experiments on CIFAR-10 show that, our approach can significantly improve stability compared to the state-of-the-art, and experiments on household safety inspection show that our method accelerate the convergence of the global model. Finally, we deploy SWATM for predictive maintenance of equipment with a combined F1-score improvement of 2.57%.
‘Skimming-Perusal’ Detection: A Simple Object Detection Baseline in GigaPixel-level Images

Object detection has achieved amazing performance in regular-sized images, but with the emergence of gigapixel-level images, even the most advanced object detection methods cannot be directly used to process them quickly and efficiently. Therefore, this paper proposes a simple baseline for gigapixel-level images object detection called Skimming-Perusal Detection (SPDet). The SPDet consists mainly of two parts, a skimming model and a perusal model. The skimming model is based on an efficient global-to-local search strategy to detect possible regions containing objects. Non-object regions are merged through a skimming iterative merging strategy to generate skimming patch candidates. The perusal model adaptive scales the skimming patch candidates guided by the coarse detection of the skimming model. Extensive evaluations on the PANDA dataset demonstrate that the SPDet boosts detection speed on gigapixel-level images by 6× while achieving better performance than a variety of state-of-the-art methods. The source code is released at https://github.com/TJUT-CV/SPDet.
Cluster-driven GNN-based Federated Recommendation with Biased Message Dropout

Due to the remarkable ability to model the high-order links within user-item relations, the graph neural network (GNN) is gradually applied to personalized recommendations in many online services. Besides, federated learning (FL) recently emerged as a powerful framework that enables collaborative training while protecting user data privacy. However, the integration of GNN and FL still exists with vital challenges unsolved, e.g., learning from non-IID local sub-graphs with only low-order user-item interactions jointly and overcoming the over-fitting problems with high training efficiency. In this paper, we propose CdFed, a Cluster-driven GNN-based Federated Learning framework, to address the GNN+FL challenges. CdFedhas two major components. First, to learn from non-IID sub-graphs, we design an Adaptive Model Clustering (AMC) strategy that takes advantage of the similarity across the uploaded model weights and updates clusters adaptively in each communication round. Second, we develop a Biased Message Dropout (BMD) strategy to combat the overfitting problem and accelerate the training process of federated learning. Together with AMC and BMD, CdFedcan implicitly complete the missing links between sub-graphs more efficiently and greatly improve the model’s generalization ability in non-IID scenarios. We have conducted extensive evaluations and the results reveal that our proposed approaches can outperform the SOTA solution by 24% in model performance and 8x in training speed.
Showcasing Native Wildflowers in Virtual Reality

The goal of this project is to develop a virtual reality program on the Oculus Quest 2 to showcase and educate users on native wildflowers that can be found in North America, specifically in the United States. We have created a virtual environment with framed images of some of America's native flowers and, using interactive menus, we provide educational information on each plant. In addition to the 2D images, a 360-image viewer was implemented to show some of the native habitats of these plants to create a more immersive learning experience. Using the Oculus Quest's experimental hand tracking feature, users can switch between using the Oculus controller and using their actual hands as the controller, allowing the users to choose how they interact with the virtual environment. The United States is home to many beautiful native flowers and by showcasing them in virtual reality anyone can experience these flowers and learn about them.
Image Protection for Robust Cropping Localization and Recovery

Existing image cropping detection schemes ignore that recovering the cropped-out contents can unveil the purpose of the behaved cropping attack. This paper presents CLR-Net, a novel image protection scheme addressing the combined challenge of image Cropping Localization and Recovery. We first protect the original image by introducing imperceptible perturbations. Then, typical image post-processing attacks are simulated to erode the protected image. On the recipient’s side, we predict the cropping mask and recover the original image. Besides, we propose a novel Fine-Grained generative JPEG simulator (FG-JPEG) as well as a feature alignment network to improve the real-world robustness. Comprehensive experiments prove that the quality of the recovered image and the accuracy of crop localization are both satisfactory.
Optimizing Video Streaming for Sustainability and Quality: The Role of Preset Selection in Per-Title Encoding

HTTP Adaptive Streaming (HAS) methods divide a video into smaller segments, encoded at multiple pre-defined bitrates to construct a bitrate ladder. Bitrate ladders are usually optimized per title over several dimensions, such as bitrate, resolution, and framerate. This paper adds a new dimension to the bitrate ladder by considering the energy consumption of the encoding process. Video encoders often have multiple pre-defined presets to balance the trade-off between encoding time, energy consumption, and compression efficiency. Faster presets disable certain coding tools defined by the codec to reduce the encoding time at the cost of reduced compression efficiency. Firstly, this paper evaluates the energy consumption and compression efficiency of different x265 presets for 500 video sequences. Secondly, optimized presets are selected for various representations in a bitrate ladder based on the results to guarantee a minimal drop in video quality while saving energy. Finally, a new per title model, which optimizes the trade-off between compression efficiency and energy consumption, is proposed. The experimental results show that decreasing the VMAF score by 0.15 and 0.39 while choosing an optimized preset results in encoding energy savings of 70% and 83%, respectively.
Iterative Refinement Network for Hyperspectral Image Denoising

Hyperspectral image (HSI) denoising is an important pre-processing procedure for subsequent tasks. Learning a direct mapping from the observed noisy HSI to its clean counterpart is challenging, especially in the case of very severe noise. The learning difficulty can be greatly reduced with the iterative refinement, combining the denoising results with the noisy HSI to produce a cleaner HSI for further denoising. To this end, we introduce an iterative refinement denoising network (IRDNet) for HSIs. The network consists of three key components, i.e., a coarse estimation module, a multi-stage refinement module, and a λ(•) module. The coarse estimation module provides the initial estimate for the starting point of further refinement. The lightweight refinement module progressively performs noise reduction on the weighted combination of noisy inputs and denoising results from the previous layer. Instead of fixed weights, the λ(•) module adaptively provides the layer-wise weight for each band for the aforementioned combinations. Extensive experiments on synthetic and real-world datasets show that our IRDNet favorably outperforms alternative methods.
Multi-Level Correlation Network For Few-Shot Image Classification

Few-shot image classification(FSIC) aims to recognize novel classes given few labeled images from base classes. Recent works have achieved promising classification performance, especially for metric-learning methods, where a measure at only image feature level is usually used. In this paper, we argue that measure at such a level may not be effective enough to generalize from base to novel classes when using only a few images. Instead, a multi-level descriptor of an image is taken for consideration in this paper. We propose a multi-level correlation network (MLCN) for FSIC to tackle this problem by effectively capturing local information. Concretely, we present the self-correlation module and cross-correlation module to learn the semantic correspondence relation of local information based on learned representations. Moreover, we propose a pattern-correlation module to capture the pattern of fine-grained images and find relevant structural patterns between base classes and novel classes. Extensive experiments and analysis show the effectiveness of our proposed method on four widely-used FSIC benchmarks.
Leveraging Attribute Knowledge for Open-set Action Recognition

Open-set action recognition(OSAR) aims to recognize known classes and reject unknown classes. Most OSAR methods focus on learning a favorable threshold to distinguish known and unknown samples in a pure data-driven manner. However, these methods do not utilize the prior knowledge of action classes. In this paper, we propose to Leverage Attribute Knowledge (LAK) for OSAR. Specifically, the class-attribute knowledge learning is designed to integrate attribute knowledge into the model based on spatial-temporal features. Here, attributes are used as a bridge, linking known and unknown classes implicitly to make up the knowledge gap. Furthermore, a learnable relation matrix is adaptively adjusted during training to obtain the class-attribute relations that are expected to be generalized in open-set settings. Extensive experiments on three popular datasets show that the proposed method achieves state-of-the-art performance.
Multi-Scale Query-Adaptive Convolution for Generalizable Person Re-Identification

Domain Generalization in person re-identification (ReID) aims to learn a generalizable model from a single or multi-source domain that can be directly deployed to an unseen domain without fine-tuning. In this paper, we investigate the problem of single-source domain generalization in ReID. Recent research has gained remarkable progress by treating image matching as a search for local correspondences in feature maps. However, to ensure efficient matching, they usually adopt a pixel-wise matching approach, which is prone to be deviated by the identity-irrelevant patch features in the image, such as background patches. To address this problem, we propose the Multi-scale Query-Adaptive Convolution (QAConv-MS) framework. Specifically, we adopt a group of template kernels with different scales to extract local features of different receptive fields from the original feature maps and accordingly perform the local matching process. We also introduce a self-attention branch to extract global features from the feature map as complementary information for local features. Our approach achieves state-of-the-art performances on four large-scale datasets.
Co-designing Immersive and Inclusive Virtual Museum with children and People with Disabilities: a Pilot Study

Immersive virtual environments represent a great opportunity for museums to enhance visitor experience through edutainment. However, to provide an enjoyable entertainment and learning experience for all visitors, including people with disabilities, the virtual museum must not only be accessible, but also inclusive: they must provide greater equality and cultural and learning opportunities for all social groups. To achieve this goal, the concept of Universal Design needs to evolve into a user-centered approach where people are involved in co-designing the virtual museum experience. In this context, the article describes a pilot study conducted at the University of Macerata, which explores the possibility of using high-fidelity prototyping in a virtual laboratory to support the co-creation of an immersive virtual museum environment with relevant target users, including children and people with disability, from the earliest design stages. The paper presents the results of the co-design process and discusses its implications in defining design requirements to ensure the accessibility of immersive solutions for cultural heritage.
Cross-Modality Fourier Feature for Medical Image Synthesis

In this paper, we propose a cross-modality fourier feature (CMFF) method via frequency selection, which learns the rational anatomical structure for targeting medical modality images. Unlike existing works seeking pixel-wise intensity discrepancy likely misleading to bias anatomical structures, our approach strongly holds the medical prior that different modality MRI images should share the same anatomical structure. To achieve this, our method instead learns to convert MRI images to the auxiliary frequency domain. Moreover, we adopt the Shapley value to quantify the contribution of each frequency, with respect to the structure for MRI image pairs of different modalities. Thus our approach learns to refine the anatomical structure generated by the target modality iteratively. Extensive experimental results on the BraTS dataset show that our model surpasses the performance compared to SOTAs.
Hierarchical Graph Neural Network for Human Pose Estimation

The human bodies are hierarchical structure and human pose estimation is highly dependent on the correlation between the keypoints of human body. Most existing CNN-based and Transformer-based methods perform well in the visual representation but lack the ability to learn the correlation between keypoints explicitly. To address this problem, we propose a human pose estimation method based on a hierarchical graph neural network with dynamic keypoint weight. The hierarchical graph neural network consists of three parts: the inter-layer process, hierarchical process, and iterative inference. The inter-layer process and hierarchical process learn the correlation between similar keypoints and further keypoints, respectively. And iterative inference is used to refine and aggregate more useful information between keypoints. Furthermore, the dynamic keypoint weight network gives each keypoint a different degree of importance. Extensive experiments demonstrate that our HGNPose-M and HGNPose-L achieve 76.5 AP $(\color{Red}{\uparrow 2.1})$ and 77.4 AP $(\color{Red}{\uparrow 1.6})$ on the COCO dataset respectively.
A Process Model for Ensuring Diffusion and Adhesion of Cybersecurity Innovations in Software Development Organizations

Optimizing existing business processes through centralization and uniformity of solutions is the preferred strategy for cyber risk reduction. However, the analysis of cybersecurity needs that reflect the objectives, design, and architecture of the respective missions or products is necessary to ensure the best alignment of cybersecurity innovations with various facets of the overarching organizational objectives. By employing the design science methodology, this study developed an artifact that incorporated the closed-loop Innovation Adhesion Evaluation Process (IAEP) to be utilized by subject matter experts (SME) in assessing the feasibility, timeliness, suitability, and sustainability of cybersecurity-related innovations during the operational phase of a project life cycle. The study found that a group of SMEs can use the IAEP model to evaluate the proposed innovation's potential impact on missions or products.
MBDFNet: Multi-scale Bidirectional Dynamic Feature Fusion Network for Efficient Image Deblurring

Existing deep image deblurring models achieve favorable results with growing model complexity. However, these models cannot be applied to those low-power devices with resource constraints (e.g., smart phones) as these models usually have lots of network parameters and require computational costs. To overcome this problem, we develop a multi-scale bidirectional dynamic feature fusion network (MBDFNet), a lightweight deep deblurring model, for efficient image deblurring. The proposed MBDFNet progressively restores multi-scale latent clear images from blurry input based on a multi-scale framework. To better utilize the features from coarse scales, we propose a bidirectional gated dynamic fusion module so that the most useful information of the features from coarse scales are kept to facilitate the estimations in the finer scales. We solve the proposed MBDFNet in an end-to-end manner and show that it has fewer network parameters and lower FLOPs values, where the FLOPs value of the proposed MBDFNet is at least 6× smaller than the state-of-the-art methods. Both quantitative and qualitative evaluations show that the proposed MBDFNet achieves favorable performance in terms of model complexity while having competitive performance in terms of accuracy against state-of-the-art methods.
Interpretable Generative Modeling Using a Hierarchical Topological VAE

Generating realistic datasets with fine-grained control over their properties can help overcome challenges linked to the scarcity of data in many domains, such as medical applications. To that end, we extend Variational Autoencoders by using a hierarchical and topological prior consisting of a sequence of Self-Organizing Maps (SOM), which are stacked in the latent space and learned without supervision, jointly with the parameters of the variational autoencoder. We induce a hierarchy between the codes of the SOM sequence, each SOM corresponding to a different hierarchical level and learning increasingly finer-grained representations of the data. Our model combines the power of deep learning with the interpretability of hierarchical and topological clustering and produces competitive results when evaluated on three well-known computer vision benchmarks and a custom medical dataset.
Preserving Locality in Vision Transformers for Class Incremental Learning

Learning new classes without forgetting is crucial for real-world applications for a classification model. Vision Transformers (ViT) recently achieve remarkable performance in Class Incremental Learning (CIL). Previous works mainly focus on block design and model expansion for ViTs. However, in this paper, we find that when the ViT is incrementally trained, the attention layers gradually lose concentration on local features. We call this interesting phenomenon as Locality Degradation in ViTs for CIL. Since the low-level local information is crucial to the transferability of the representation, it is beneficial to preserve the locality in attention layers. In this paper, we encourage the model to preserve more local information as the training procedure goes on and devise a Locality-Preserved Attention (LPA) layer to emphasize the importance of local features. Specifically, we incorporate the local information directly into the vanilla attention and control the initial gradients of the vanilla attention by weighting it with a small initial value. Extensive experiments show that the representations facilitated by LPA capture more low-level general information which is easier to transfer to follow-up tasks. The improved model gets consistently better performance on CIFAR100 and ImageNet100. The source code is available at https://github.com/bwnzheng/LPA_ICME2023.