Reviewerly/exHarmony huggingface dataset:
Dynamic Transformer for Few-shot Instance Segmentation

Few-shot instance segmentation aims to train an instance segmentation model that can fast adapt to novel classes with only a few reference images. Existing methods are usually derived from standard detection models and tackle few-shot instance segmentation indirectly by conducting classification, box regression, and mask prediction on a large set of redundant proposals followed by indispensable post-processing, e.g., Non-Maximum Suppression. Such complicated hand-crafted procedures and hyperparameters lead to degraded optimization and insufficient generalization ability. In this work, we propose an end-to-end Dynamic Transformer Network, DTN for short, to directly segment all target object instances from arbitrary categories given by reference images, relieving the requirements of dense proposal generation and post-processing. Specifically, a small set of Dynamic Queries, conditioned on reference images, are exclusively assigned to target object instances and generate all the instance segmentation masks of reference categories simultaneously. Moreover, a Semantic-induced Transformer Decoder is introduced to constrain the cross-attention between dynamic queries and target images within the pixels of the reference category, which suppresses the noisy interaction with the background and irrelevant categories. Extensive experiments are conducted on the COCO-20 dataset. The experiment results demonstrate that our proposed Dynamic Transformer Network significantly outperforms the state-of-the-arts.
Towards Continual Adaptation in Industrial Anomaly Detection

Anomaly detection (AD) has gained widespread attention due to its ability to identify defects in industrial scenarios using only normal samples. Although traditional AD methods achieved acceptable performance, they mainly focus on the current set of examples solely, leading to catastrophic forgetting of previously learned tasks when trained on a new one. Due to the limitation of flexibility and the requirements of realistic industrial scenarios, it is urgent to enhance the ability of continual adaptation of AD models. Therefore, this paper proposes a unified framework by incorporating continual learning (CL) to achieve our newly designed task of continual anomaly detection (CAD). Note that, we observe that data augmentation strategy can make AD methods well adapted to supervised CL (SCL) via constructing anomaly samples. Based on this, we hence propose a novel method named Distribution of Normal Embeddings (DNE), which utilizes the feature distribution of normal training samples from past tasks. It not only effectively alleviates catastrophic forgetting in CAD but also can be integrated with SCL methods to further improve their performance. Extensive experiments and visualization results on the popular benchmark dataset MVTec AD, have demonstrated advanced performance and the excellent continual adaption ability of our proposed method compared to other AD methods. To the best of our knowledge, we are the first to introduce and tackle the task of CAD. We believe that the proposed task and benchmark will be beneficial to the field of AD. Our code is available in thesupplementary material.
Region-based Pixels Integration Mechanism for Weakly Supervised Semantic Segmentation

Image-level annotations allow to achieve semantic segmentation in a weakly-supervised way. Most advanced approaches utilize class activation map (CAM) from deep classifier to generate pseudo-labels. However, CAM generally only focuses on the most discriminative parts of targets. To explore more pixel-level semantic information and recognize all pixels within the objects for segmentation, we propose a Region-based Pixels Integration Mechanism (RPIM) which discovers the intra-region and inter-region information. Firstly, the foreground regions are formed on the basis of superpixels and the initial responses. Each region is regarded as a subtree, whose nodes are the image pixels within the region. Then, an Intra-region Integration (IRI) Module is designed to explore the nodes relationships inside the subtree. Within each subtree, nodes will vote for the most confident class and share the highest probability. Moreover, an Inter-region Spreading (IRS) Module is proposed to further improve the consistency of CAM. For each class, the most confident unprocessed subtree finds their homologous neighbors, connects with them and shares its probability. By iterative refinement, the training process will integrate the individual nodes into region subtrees, and gradually form the subtrees with similar probabilities to the object semantic trees for each foreground class. To our best knowledge, our approach achieves the state-of-the-art performance on PASCAL VOC 2012 validation set with 71.4% mIoU. The experiments also show that our scheme is plug-and-play and can collaborate with different approaches to improve their performance.
AI-VQA

Visual Question Answering (VQA) serves as a proxy for evaluating the scene understanding of an intelligent agent by answering questions about images. Most VQA benchmarks to date are focused on those questions that can be answered through understanding visual content in the scene, such as simple counting, visual attributes, and even a little challenging questions that require extra encyclopedic knowledge. However, humans have a remarkable capacity to reason dynamic interaction on the scene, which is beyond the literal content of an image and has not been investigated so far. In this paper, we propose Agent Interaction Visual Question Answering (AI-VQA), a task investigating deep scene understanding if the agent takes a certain action. For this task, a model not only needs to answer action-related questions but also to locate the objects in which the interaction occurs for guaranteeing it truly comprehends the action. Accordingly, we make a new dataset based on Visual Genome and ATOMIC knowledge graph, including more than 19,000 manually annotated questions, and will make it publicly available. Besides, we also provide an annotation of the reasoning path while developing the answer for each question. Based on the dataset, we further propose a novel method, called ARE, that can comprehend the interaction and explain the reason based on a given event knowledge base. Experimental results show that our proposed method outperforms the baseline by a clear margin.
CLIPTexture: Text-Driven Texture Synthesis

Can artificial intelligence create textures with artistic value according to human language control? Existing texture synthesis methods require example texture input. However, in many practical situations, users don't have satisfying textures but tell designers about their needs through simple sketches and verbal descriptions. This paper proposes a novel texture synthesis framework based on the CLIP, which models the texture synthesis problem as an optimization process and realizes text-driven texture synthesis by minimizing the distance between the input image and the text prompt in latent space. Our method performs zero-shot image manipulation successfully even between unseen domains. We implement texture synthesis using two different optimization methods, the TextureNet and Diffvg, demonstrating the generality of CLIPTexture. Extensive experiments confirmed the robust and superior manipulation performance of our methods compared to the existing baselines.
Machine Unlearning for Image Retrieval

Data owners have the right to request for deleting their data from a machine learning (ML) model. In response, a naïve way is to retrain the model with the original dataset excluding the data to forget, which is however unrealistic as the required dataset may no longer be available and the retraining process is usually computationally expensive. To cope with this reality, machine unlearning has recently attained much attention, which aims to enable data removal from a trained ML model responding to deletion requests, without retraining the model from scratch or full access to the original training dataset. Existing unlearning methods mainly focus on handling conventional ML methods, while unlearning deep neural networks (DNNs) based models remains underexplored, especially for the ones trained on large-scale datasets.
Adaptive Camera Margin for Mask-guided Domain Adaptive Person Re-identification

Research on transferring the learned person re-identification (ReID) model in the source domain to other domains is of great importance since deploying a ReID model to a new scenario is common in practical applications. Most of existing unsupervised domain adaptation methods for person ReID employ the framework of pre-training in the source domain, and clustering and fine-tuning in the target domain. However, how to reduce the intra-domain variations and narrow the inter-domain gaps is far from solved and remains a challenging problem under this framework. In this paper, we address these issues from two aspects. Firstly, a voted-mask guided image channel shuffling strategy for data augmentation is proposed to enhance visual diversity, where image channel shuffling is used as an efficient tool to bridge the inter-domain gap, and voted masks are employed to extract the foregrounds of pedestrian images to relief the negative effects of various backgrounds for reducing the intra-domain variations. Secondly, a novel plug-and-play metric named adaptive camera margin is proposed to fully exploit the low-cost camera tags for producing high-quality pseudo labels, which can significantly reduce the intra-domain variations without extra training cost. Specifically, the proposed network consists of a sensitive branch and an adaptive branch accompanied with our strategy of data augmentation, which are embedded into a joint learning framework to decouple visual representations for better capturing transferable features across different domains in both two stages. Adaptive camera margin is employed to pull samples with different camera IDs closer in the procedure of DBSCAN clustering, which can reduce the influence of intra-domain variations caused by camera shift to a large extent in an effective and efficient manner. Comprehensive experiments show that the proposed method achieves competitive performance compared with state-of-the-art methods on the benchmark datasets. Source code will be released at: https://github.com/ahuwangrui/MACM.
Defending Physical Adversarial Attack on Object Detection via Adversarial Patch-Feature Energy

Object detection plays an important role in security-critical systems such as autonomous vehicles but has shown to be vulnerable to adversarial patch attacks. Existing defense methods are restricted to localized noise patches by removing noisy regions in the input image. However, adversarial patches have developed into natural-looking patterns which evade existing defenses. To address this issue, we propose a defense method based on a novel concept "Adversarial Patch- Feature Energy" (APE) which exploits common deep feature characteristics of an adversarial patch. Our proposed defense consists of APE-masking and APE-refinement which can be employed to defend against any adversarial patch on literature. Extensive experiments demonstrate that APE-based defense achieves impressive robustness against adversarial patches both in the digital space and the physical world.
Gaze- and Spacing-flow Unveil Intentions: Hidden Follower Discovery

We raise a new and challenging multimedia application in video surveillance system, i.e., Hidden Follower Discovery (HFD). In contrast to the common abnormal behaviors that are occurring, hidden following is not an ongoing activity, but a preparatory action. Hidden following behavior does not have salient features, making it hard to be discovered. Fortunately, from a socio-cognitive perspective, we found and verified the phenomena that the gaze-flow pattern and the spacing-flow pattern between hidden and normal followers are different. To promote HFD research, we construct two pioneering datasets and devise an HFD baseline network based on the recognition of both gaze-flow and spacing-flow patterns from surveillance videos. Extensive experiments demonstrate their effectiveness.
Geometry-Aware Reference Synthesis for Multi-View Image Super-Resolution

Recent multi-view multimedia applications struggle between high-resolution (HR) visual experience and storage or bandwidth constraints. Therefore, this paper proposes a Multi-View Image Super-Resolution (MVISR) task. It aims to increase the resolution of multi-view images captured from the same scene. One solution is to apply image or video super-resolution (SR) methods to reconstruct HR results from the low-resolution (LR) input view. However, these methods cannot handle large-angle transformations between views and leverage information in all multi-view images. To address these problems, we propose the MVSRnet, which uses geometry information to extract sharp details from all LR multi-view to support the SR of the LR input view. Specifically, the proposed Geometry-Aware Reference Synthesis module in MVSRnet uses geometry information and all multi-view LR images to synthesize pixel-aligned HR reference images. Then, the proposed Dynamic High-Frequency Search network fully exploits the high-frequency textural details in reference images for SR. Extensive experiments on several benchmarks show that our method significantly improves over the state-of-the-art approaches.
A Preliminary Earthquake Detection from The Seismic Temporary Network in The Opak Fault

Abstract Microseismic observation is a mandatory tool for analysing and imaging the progress of source earthquake mechanism. The observation of low-quality signal is usually found because of false detection, transient signals by natural noise, or related to seismometer condition and some human activities. Therefore, we try to figure the microseismic phenomenon after major earthquake in the Jogjakarta fault that was generated by an active fault namely Opak Fault. We used the seismic data recording from the seismic project that was installed in 2006 cover the Jogjakarta region. We used one week data to see the potential detection and highlight the possible of positive or negative false detection. With total 16 stations, we got ~500 events in only one week recording. We used a various threshold with 110 to scan all the dataset and we obtained ~450 events with 50-100 events with possible of false positives. Meanwhile, the threshold of 112 has &lt; 50 events that could be suitable with the waveforms. The example results of 120 thresholds figure a strong event that is located by highly resolution of stack coherent from some stations with precise of P and S phase fitting. This study present the first automatic earthquake locations that can provide more detail of seismic structure information in the Jogjakarta region.
AdsCVLR: Commercial Visual-Linguistic Representation Modeling in Sponsored Search

Sponsored search advertisements (ads) appear next to search results when consumers look for products and services on search engines. As the fundamental basis of search ads, relevance modeling has attracted increasing attention due to the significant research challenges and tremendous practical value. In this paper, we address the problem of multi-modal modeling in sponsored search, which models the relevance between user query and commercial ads with multi-modal structured information. To solve this problem, we propose a transformer architecture with Ads data on Commercial Visual-Linguistic Representation (AdsCVLR) with contrastive learning that naturally extends the transformer encoder with the complementary multi-modal inputs, serving as a strong aggregator of image-text features. We also make a public advertising dataset, which includes 480K labeled query-ad pairwise data with structured information of image, title, seller, description, and so on. Empirically, we evaluate the AdsCVLR model over the large industry dataset, and the experimental results of online/offline tests show the superiority of our method.
Augmented Objects as Portals into Virtual Worlds:Using Audio to Create Immersive Experiences in Extended Realities

Building on a long history of the integration of new technology into the arts, in this research we set out to develop and evaluate new methods for creating immersive, audio-centric artworks in a Mixed Reality (MR) context. We employ multiple levels of spatial audio, both within the room as well as the umbrellas that participants navigate the space with, in order to provide a 6 degree of freedom immersive experience. Furthermore, we use dynamic projection mapping, again both throughout the global environment, and on the umbrellas, to create immersive experiences. This allows the umbrellas to function as so-called Augmented Objects that facilitate the composition and navigation of Responsive Sonic Environments, thus encouraging participants to examine their own perceptions of reality while incorporating them into the larger artistic installation. Additionally, we explore the creation of Spatial Metacompositions, navigable environments that function as performance spaces for the participants who interact with them. Finally, we present the technical foundations necessary to create such systems and discuss techniques to effectively employ them in one’s artistic practice.
A Parameter-free Multi-view Information Bottleneck Clustering Method by Cross-view Weighting

With the fast-growing multi-modal/media data in the Big Data era, multi-view clustering (MVC) has attracted lots of attentions lately. Most MVCs focus on integrating and utilizing the complementary information among views by linear sum of the learned view weights and have shown great success in some fields. However, they fail to quantify how complementary the information across views actually utilized for benefiting final clustering. Additionally, most of them contain at least one parameter for regularization without prior knowledge, which puts pressure on the parameter-tuning and thus makes them impractical. In this paper, we propose a novel parameter-free multi-view information bottleneck (PMIB) clustering method to automatically identify and exploit useful complementary information among views, thus reducing the negative impact from the harmful views. Specifically, we first discover the informative view by measuring the relevant information preserved by the original data and the compact clusters with mutual information. Then, a new cross-view weight learning scheme is designed to learn how complementary between the informative view and remaining views. Finally, the quantitative correlations among views are fully exploited to improve the clustering performance without needing any additional parameters or prior knowledge. Experimental results on different kinds of multi-view datasets show the effectiveness of the proposed method.
Image Road Surface Classification Based on GLCM Feature Using LGBM Classifier

Abstract The road is a land transportation infrastructure that is often used by the community to support their daily activities. Damaged road conditions will cause inconvenience and especially safety for users. Therefore, road repairs must be carried out immediately. The problem currently being faced by the public works services is that identifying road damage is still done manually, so it takes a long time and costs, therefore the approach can be taken is using a computer to do texture feature extraction on road surface images to obtain information using the GLCM method and classify using the LGBM method for normal, cracked, potholed road surfaces with an accuracy of 90%.
Multi-Scale Coarse-to-Fine Transformer for Frame Interpolation

The majority of prevailing video interpolation methods compute flows to estimate the intermediate motion. However, accurate estimation of the intermediate motion is difficult with low-order motion model hypothesis, which induces enormous difficulties for subsequent processing. To alleviate the limitation, we propose a two-stage flow-free video interpolation architecture. Rather than utilizing pre-defined motion models, our method represents complex motion through data-driven learning. In the first stage, we analyze spatial-temporal information and generate coarse anchor frame features. In the second stage, we employ transformers to transfer neighboring features to the intermediate time steps and enhance the spatial textures. To improve the quality of coarse anchor frame features and the robustness in dealing with the multi-scale textures with large-scale motion, we propose a multi-scale architecture and transformers with variable token sizes to progressively enhance the features. The experimental results demonstrate that our model outperforms state-of-the-art methods for both single frame and multi frames interpolation tasks, and the extended ablation studies verify the effectiveness of our model.
D<sup>2</sup>ADet: dual-activation domain RetinaNet based on a deformable feature pyramid network for ship detection in SAR images

RetinaNet has been widely used in the field of ship detection in synthetic aperture radar (SAR) images due to its simple structure and excellent performance. Although various modified RetinaNet models for SAR ship detection have been proposed, there are still some problems, such as how to capture more complete ship features and the unbalanced distribution of positive samples in different tasks, which have not been mentioned in previous studies. Aiming at the above problems, this paper proposes a modified RetinaNet detector for SAR ship detection, namely D 2ADet. Specifically, a modified feature pyramid network is proposed to capture adaptive features that are more suitable for the ship. Then, a dual-activation domain training method is designed to decouple the sample assignment processes of different tasks to meet their requirements of positive samples for each task. Experimental results show that compared with RetinaNet, D 2ADet obtains 7.67%, 11.57% and 7.28% gains in AP50, AP75 and mAP on the public SAR ship detection dataset (SSDD), respectively. And D 2ADet achieves the state-of-the-art performance compared with other methods.
Deep Video Understanding with a Unified Multi-Modal Retrieval Framework

In this paper, we propose a unified multi-modal retrieval framework to tackle two typical video understanding tasks, i.e., matching movie scenes and text descriptions, and scene sentiment classification. For the task of matching movie scenes and text descriptions, it is a natural multi-modal retrieval problem, while for the task of scene sentiment classification, the proposed framework aims at finding most related sentiment tag for each movie scene, which is also a multi-modal retrieval problem. By considering these two tasks as multi-modal retrieval problems, we propose a unified multi-modal retrieval framework, which can make full use of the models pre-trained on large scale multi-modal datasets, experiments show that it is critical for the tasks which have only hundreds of training examples. To further improve the performance on movie video understanding task, we also collect a large scale video-text dataset, which contains 427,603 movie-shot and text pairs. Experimental results validate the effectiveness of this dataset.