Reviewerly/exHarmony huggingface dataset:
Prototype Queue Learning for Multi-Class Few-Shot Semantic Segmentation

Few-shot semantic segmentation aims to undertake the segmentation task of novel classes with only a few annotated images. However, most existing methods tend to segment the foreground and background in the image, which limits practical application. In this paper, we present a Prototype Queue Network, which performs few-shot segmentation on multiclass in the images by aggregating binary classes into multiple classes. A prototype queue learning module is proposed to achieve multi-class segmentation by mining the relationship among features of different classes with queue and pseudo labels. In addition, a background latent class distribution refinement module is proposed to prevent the latent novel class in the background from being incorrectly predicted, which refines the boundary among different classes. Furthermore, we propose a two-steps segmentation module to optimize the process of extracting feature representation by adding progressive constraints, which can further improve the accuracy of segmentation. Experiments on the UDD and Vaihingen datasets demonstrate that our method achieves state-of-the-art performance.
Relational Future Captioning Model for Explaining Likely Collisions in Daily Tasks

Domestic service robots that support daily tasks are a promising solution for elderly or disabled people. It is crucial for domestic service robots to explain the collision risk before they perform actions. In this paper, our aim is to generate a caption about a future event. We propose the Relational Future Captioning Model (RFCM), a crossmodal language generation model for the future captioning task. The RFCM has the Relational Self-Attention Encoder to extract the relationships between events more effectively than the conventional self-attention in transformers. We conducted comparison experiments, and the results show the RFCM outperforms a baseline method on two datasets.
Conditional Reconstruction for Open-Set Semantic Segmentation

Open set segmentation is a relatively new and unexplored task, with just a handful of methods proposed to model such tasks. We propose a novel method called CoReSeg that tackles the issue using class conditioned reconstruction of the input images according to their pixelwise mask. Our method conditions each input pixel to all known classes, expecting higher errors for pixels of unknown classes. It was observed that the proposed method produces better semantic consistency in its predictions than the baselines, resulting in cleaner segmentation maps that better fit object boundaries. CoReSeg outperforms state-of-the-art methods on the Vaihingen and Potsdam ISPRS datasets, while also being competitive on the Houston 2018 IEEE GRSS Data Fusion dataset. Our official implementation for CoReSeg is available at: https://github.com/iannunes/CoReSeg.
Reduced Dependency Fast Unsupervised 3D Face Reconstruction

Recent 3D face reconstruction methods show encouraging results in retrieving 3D face shape and texture from monocular face images. However, these approaches pose several dependencies during testing, such as the requirement of facial landmark coordinates. Moreover, a large testing time presents a challenge for real-time applications. To address these issues, we propose REduced Dependency Fast UnsuperviSEd 3D Face Reconstruction (RED-FUSE) framework, which uses unprocessed face images to estimate reliable 3D face shape and texture, thus eliminating the need for prior land-mark knowledge, and considerable prediction time during testing. RED-FUSE outperforms the current state-of-the-art method on CelebA dataset e.g., for 3D shape and color-based errors, a reduction from 5.84 ± 0.16 to 3.14 ± 0.11 and from 3.50 ± 0.14 to 2.97 ± 0.09 is observed, leading to an improvement of 46.2% and 15.1%, respectively. In addition, the testing time reduces from 7.30 msec to 1.85 msec per face, showing the effectiveness of our method.
Measurably Stronger Explanation Reliability Via Model Canonization

While rule-based attribution methods have proven useful for providing local explanations for Deep Neural Networks, explaining modern and more varied network architectures yields new challenges in generating trustworthy explanations, since the established rule sets might not be sufficient or applicable to novel network structures. As an elegant solution to the above issue, network canonization has recently been introduced. This procedure leverages the implementation-dependency of rule-based attributions and restructures a model into a functionally identical equivalent of alternative design to which established attribution rules can be applied. However, the idea of canonization and its usefulness have so far only been explored qualitatively. In this work, we quantitatively verify the beneficial effects of network canonization to rule-based attributions on VGG-16 and ResNet18 models with BatchNorm layers and thus extend the current best practices for obtaining reliable neural network explanations.
Multi-Scale Deformable Transformer Encoder Based Single-Stage Pedestrian Detection

Pedestrian detection is a key task in intelligent video surveillance systems which requires both fast inference and high detection accuracy. Although single-stage deep learning pedestrian detectors have achieved relatively high detection accuracy with simpler architecture and less inference time, their performance is limited compared to two-stage methods. The reason is the lack of scale-aware features without the assistance of proposal regions. To overcome this, a multi-scale deformable transformer encoder-based module is proposed. It can extract the sparse important features at deformable sampling locations from multiple levels. The proposed architecture significantly improves the performance compared to the baseline center and scale prediction method on both Caltech and Citypersons datasets. It even outperforms the state-of-the-art two-stage methods in detecting heavily occluded pedestrians on Citypersons validation set.
Maskformer with Improved Encoder-Decoder Module for Semantic Segmentation of Fine-Resolution Remote Sensing Images

In 2021, the Transformer based models have demonstrated extraordinary achievement in the field of computer vision. Among which, Maskformer, a Transformer based model adopting the mask classification method, is an outstanding model in both semantic segmentation and instance segmentation. Considering the specific characteristics of semantic segmentation of remote sensing images (RSIs), we design CADA-MaskFormer(a Mask classification-based model with Cross-shaped window self-Attention and Densely connected feature Aggregation) based on Maskformer by improving its encoder and pixel decoder. Concretely, the mask classification that generates one or even more masks for specific category to perform the elaborate segmentation is especially suitable for handling the characteristic of large within-class and small between-class variance of RSIs. Furthermore, we apply the Cross-Shaped Window self-attention mechanism to model the long-range context information contained in RSIs at maximum extent without the increasing of computational complexity. In addition, the Densely Connected Feature Aggregation Module (DCFAM) is used as the pixel decoder to incorporate multi-level feature maps from the encoder to get a finer semantic segmentation map. Extensive experiments conducted on two remotely sensed semantic segmentation datasets Potsdam and Vaihingen achieves 91.88% and 91.01% in OA index respectively, outperforming most of competitive models designed for RSIs. The code is available from https://github.com/lqwrl542293/JL-Yang_CV/tree/master/CADA_Maskformer
Iterative Contrastive Learning for Single Image Raindrop Removal

Deep learning has achieved remarkable progress in computer vision and image analysis. However, raindrop removal from single image still remains challenging, due to a wide range of raindrop diversities and surface reflections. In this paper, we propose an iterative neural network with feedback strategy and contrastive learning for single image raindrop removal. First, we design an iterative feedback neural network to refine low-level representations with high-level information, i.e., the output of the previous iteration is used as input for the next iteration, together with the input image with raindrops. As a result, raindrops could be gradually removed through this feedback manner. Then, we deploy contrastive regularization to push the restored image from each iteration close to the clean images without raindrops, but away from rainy images with raindrops. Extensive experiments on two raindrop benchmark datasets demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art methods. The methodology in this work could be further extended to self-supervised contrastive learning to obtain robust feature representations with less labelled data.
Haar Wavelet-Based Attention Network for Image Dehazing

Weather conditions such as haze, mist, and fog can degrade the image clarity in outdoor scenes. Although various image dehazing techniques exist in the literature, it is very challenging to remove non-homogeneous and/or densely concentrated homogeneous fog. We propose an end-to-end dehazing network <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> that utilizes attention-based learning from Haar wavelet coefficients. The model uses multi-scale wavelet transformation to break down feature maps into low- and high-frequency coefficients. The channel attention layer learns haze features while the spatial attention layer focuses on the feature location from these coefficients to further refine the output. Quantitative and visual analyses demonstrate that the proposed framework outperforms recent state-of-the-art methods in removing haze from dense and non-homogeneous hazy images while maintaining color accuracy.
How Sound Affects Visual Attention in Omnidirectional Videos

In this paper, we propose a new audio-visual attention dataset that records eye movement for omnidirectional videos with and without sound. We classify the videos into three types according to the number of salient objects and sound sources and analyze the impact of sound on visual attention distribution and inter-observer consistency of viewing area in different types of videos. From the quantitative and qualitative analysis, we find that visual attention will be drawn to and concentrated on the sound source with the presence of sound, especially when there are several visually salient objects and only one sound source. Also, the sound will enhance the consistency of observation areas among viewers to some extent. For more investigations on the impact of sound on visual attention and prospective audio-visual saliency model, we still need further study.
Adaptive Proxy Anchor Loss for Deep Metric Learning

Deep metric learning (or simply called metric learning) uses the deep neural network to learn the representation of images, leading to widely used in many applications, e.g. image retrieval and face recognition. In the metric learning approaches, proxy anchor takes advantage of proxy-based and pair-based approaches to enable fast convergence time and robustness to noisy labels. However, in training the proxy anchor, selecting the hyperparameter margin is important to achieve a good performance. This selection requires expertise and is time-consuming. This paper proposes a novel method to learn the margin while training the proxy anchor approach adaptively. The proposed adaptive proxy anchor simplifies the hyperparameter tuning process while advancing the proxy anchor. We achieve state of the art on three public datasets with a noticeably faster convergence time. Our code is available at https: //github.com/tks1998/Adaptive-Proxy-Anchor
An Enhanced Transferable Adversarial Attack of Scale-Invariant Methods

Scale-invariant method (SIM) is a state-of-the-art model augmentation method to improve the transferability of adversarial examples. However, we find that SIM is easily affected by the scaling operation with small scaling factors, and cannot stably enhance the transferability of the base attack. In this paper, we propose an enhanced transferable attack based on SIM. To alleviate the instability of SIM caused by the scaled copy which does not satisfy scale-invariance, we propose to ensemble logit-outputs of scale copies of the input image, rather than ensemble the gradients, to form an ensemble attack that generates transferable adversarial images from multiple models of the original CNN model. Compared with the existing ensemble methods, our method is fast yet effective and can be easily integrated into the gradient-based attacks. The experimental results show that the proposed integrated EL-NI-FGSM attack stably improves the transferability of NI-FGSM, and outperforms SI-NI-FGSM, achieving >8% higher of attack success rate for both white-box and black-box attacks on CIFAR-10.
Investigating Explainable Artificial Intelligence for MRI-based Classification of Dementia: a New Stability Criterion for Explainable Methods

Individuals diagnosed with Mild Cognitive Impairment (MCI) have shown an increased risk of developing Alzheimer’s Disease (AD). As such, early identification of dementia represents a key prognostic element, though hampered by complex disease patterns. Increasing efforts have focused on Machine Learning (ML) to build accurate classification models relying on a multitude of clinical/imaging variables. However, ML itself does not provide sensible explanations related to the model mechanism and feature contribution. Explainable Artificial Intelligence (XAI) represents the enabling technology in this framework, allowing to understand ML outcomes and derive human-understandable explanations. In this study, we aimed at exploring ML combined with MRI-based features and XAI to solve this classification problem and interpret the outcome. In particular, we propose a new method to assess the robustness of feature rankings provided by XAI methods, especially when multicollinearity exists. Our findings indicate that our method was able to disentangle the list of the informative features underlying dementia, with important implications for aiding personalized monitoring plans.
Superpixel Group-Correlation Network for Co-Saliency Detection

Co-saliency detection is a task to segment the occurring salient objects in a group of images. The biggest challenges are distracting objects in the background and ambiguity between the foreground and background. To handle these issues, we propose a novel superpixel group-correlation network (SGCN) architecture that uses a superpixel algorithm to obtain various component features from a group of images and creates a group-correlation matrix to detect the common components of those images. In this way, non-common objects can be effectively excluded from consideration, enabling a clear distinction between foreground and background. Our method outperforms current state-of-the-art methods on three popular benchmark datasets for co-saliency detection, and our extensive experiments thoroughly validate our claimed contributions.
Benchmarking Performance of Object Detection Under Image Distortions in an Uncontrolled Environment

The robustness of object detection algorithms plays a prominent role in real-world applications, especially in uncontrolled environments due to distortions during image acquisition. It has been proven that the performance of object detection methods suffers from in-capture distortions. In this study, we present a performance evaluation framework for the state-of-the-art object detection methods using a dedicated dataset containing images with various distortions at different levels of severity. Furthermore, we propose an original strategy of image distortion generation applied to the MS-COCO dataset that combines some local and global distortions to reach much better performances. We have shown that training using the proposed dataset improves the robustness of object detection by 31.5%. Finally, we provide a custom dataset including natural images distorted from MS-COCO to perform a more reliable evaluation of the robustness against common distortions. The database and the generation source codes of the different distortions are made publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1,2</sup> .
Discriminating Between Primary and Secondary Au Events in a Paragenetically Complex Archean Lode-Gold Deposit, Wawa Gold Corridor, Ontario, Canada

Abstract The Wawa gold corridor, located in the Michipicoten greenstone belt of the Superior province, Canada, comprises Au-bearing shear zones that crosscut the 2745 Ma Jubilee stock and that evolved during protracted deformation (D1-D3). Numerous generations of sulfide minerals crystallized before, during, and after these deformation events, and gold is associated with D1 arsenopyrite, D2 pyrite, and Bi-Te phases and chalcopyrite in assemblages that crosscut D3 veins. Observations of porosity and inclusions in D1 arsenopyrite and D2 pyrite suggest these sulfides underwent coupled dissolution-reprecipitation reactions. By coupling these textural observations with trace element analysis by laser ablation-inductively coupled plasma-mass spectrometry (LA-ICP-MS), it is evident that such replacement reactions generated gold from Au previously in solid solution. Conversely, textural relationships among paragenetically late gold and Bi-Te minerals are consistent with the precipitation of these phases from Bi-rich polymetallic melts. Mass balance calculations involving comparisons of the mass of Au in sulfides and the total mass of Au in their associated host rocks indicate that only D1 arsenopyrite contained enough Au to account for whole-rock Au content and overall deposit grade. Although D1 arsenopyrite is less volumetrically abundant than the various pyrite generations present in the deposit, it is often replaced by the later pyrite types, which is compatible with higher initial volumes of arsenopyrite than what is presently observed. It is concluded that the D1 Au + arsenopyrite event was the principal Au-mineralizing event in the Wawa gold corridor and that the other gold-bearing assemblages (i.e., gold + D2 pyrite, gold + Bi-Te phases + chalcopyrite) largely represent secondary mobilization of this primary enrichment. Given that LA-ICP-MS sulfide chemistry is regularly used in orogenic Au research, the approach outlined herein to assess the relative impact of distinct Au- and sulfide-mineralizing events could easily be applied to the study of other Au deposits in which complex hydrothermal parageneses are recognized.
Co-creating a conceptual AI model to promote personalized communication and engagement for people with dementia

Purpose Digital applications to promote communication and engagement for people with dementia (PWD) are gaining increasing attention for their potential to improve interpersonal relationships and quality of life (Hoel et al, 2021).The ubiquitous availability of everyday digital technologies could enable widespread use by usual carers and families and support implementation of person-centered care practices in daily care.However, a major barrier for the adoption and effectiveness of these applications is the significant time and effort required to personalize them.which may not be practically achievable for many carers.In order to address this issue, a conceptual AI framework to automate personalization of a cognitive stimulation application has been co-created with a PWD and her daughter and its feasibility investigated.Method A digital application called Aikomi Care that provides personalized stimulation programs to promote communication called was used by the mother and daughter in weekly on-line sessions for approximately 2 years.Each session used a different program manually prepared in collaboration with the daughter by selecting stimulation topics based on her mother`s life story and interests and topics which she had enjoyed in previous sessions which were arranged in a sequence form a story that could be easily understood by the mother.In addition, each session was video recorded and the behavioral response were manually annotated and correlated with the stimulation topic giving rise to the response.In this was way, a long-term Aikomi care usage data set was generated for AI model development.Results and Discussion 2 AI models were created to learn: (i) the characteristics of the stimulation topics that generate good responses by the mother in order to recommend known and new topics that could induce good responses and ii) learn the sequence order patterns of the stimulation topics in a program that result in sessions that give good overall responses.The goal was for these models to work together to provide the right content in the right sequence to optimize generation of the desired responses by the PWD.The stimulation topic recommendation algorithm uses a Word2vec (Mikolov et al, 2013) approach and the sequence prediction algorithm is based on a Long-Short-Term-Memory model (LSTM, Brownlee, 2017).Both algorithms achieved promising accuracies sufficient for further development and demonstrate how a co-creation approach can be used to develop AI models to learn and adapt stimulation programs according to the preferences and responses of a PWD.The results of the preliminary testing of the model will be presented and its potential to be used as an automated personalized storytelling function to enable intelligent augmentation of communication and engagement will be discussed.
A deep learning network for Gleason grading of prostate biopsies using EfficientNet

The most crucial part in the diagnosis of cancer is severity grading. Gleason's score is a widely used grading system for prostate cancer. Manual examination of the microscopic images and grading them is tiresome and consumes a lot of time. Hence to automate the Gleason grading process, a novel deep learning network is proposed in this work.In this work, a deep learning network for Gleason grading of prostate cancer is proposed based on EfficientNet architecture. It applies a compound scaling method to balance the dimensions of the underlying network. Also, an additional attention branch is added to EfficientNet-B7 for precise feature weighting.To the best of our knowledge, this is the first work that integrates an additional attention branch with EfficientNet architecture for Gleason grading. The proposed models were trained using H&E-stained samples from prostate cancer Tissue Microarrays (TMAs) in the Harvard Dataverse dataset.The proposed network was able to outperform the existing methods and it achieved an Kappa score of 0.5775.