Reviewerly/exHarmony huggingface dataset:
Interpretation of machine learning models using XAI - A study on health insurance dataset

This study aims to dive into the complex Machine Learning algorithms using XAI and provide explanations for outcomes that we received from these algorithms. In this paper, we have proposed to predict the cost of health insurance. The proposed work is composed of two Machine Learning algorithms, namely Multiple Linear Regression and Random Forest, followed by an explanation of predicted results using XAI. Here, we first provide a simple explanation with the help of model-specific approaches based on Microsoft's InterpretML library. Then the predicted insurance premium cost is further explained by model-agnostic techniques, LIME, and SHAP. The significance of the study is to provide a better user experience and build trust between the user and the machine. These techniques can help to check the correctness of the prediction models, as the domain experts can analyze the features that affect the outcome the most and provide their expertise.
PSDC: A Prototype-Based Shared-Dummy Classifier Model for Open-Set Domain Adaptation

Open-set domain adaptation (OSDA) aims to achieve knowledge transfer in the presence of both domain shift and label shift, which assumes that there exist additional unknown target classes not presented in the source domain. To solve the OSDA problem, most existing methods introduce an additional unknown class to the source classifier and represent the unknown target instances as a whole. However, it is unreasonable to treat all unknown target instances as a group since these unknown instances typically consist of distinct categories and distributions. It is challenging to identify all unknown instances with only one additional class. In addition, most existing methods directly introduce marginal distribution alignment to alleviate distribution shift between the source and target domains, failing to learn discriminative class boundaries in the target domain since they ignore categorical discriminative information in the adaptation. To address these problems, in this article, we propose a novel prototype-based shared-dummy classifier (PSDC) model for the OSDA. Specifically, our PSDC introduces an auxiliary dummy classifier to calibrate the source classifier and simultaneously develops a weighted adaptation procedure to align class-wise prototypes for adaptation. We further design a pseudo-unknown learning algorithm to reduce the open-set risk. Extensive experiments on Office-31, Office-Home, and VisDA datasets show that the proposed PSDC can outperform existing methods and achieve the new state-of-the-art performance. The code will be made public.
Exploring the blockchain for sustainable food supply chain

Blockchain Technology will bring a revolutionary change to the food supply chain (FSC). It will improve the visibility and traceability of the flow of food products. This study examined blockchain technology adoption in food supply chain industries in India. This study tried to examine the traceability of the food products across the food supply chain by using blockchain technology. This study will be beneficial to food industries who are thinking for a tracing and tracking mechanisms. The result shows the positivity for blockchain technology adoption in FSC industries. Also, it laid the foundation for blockchain technology adoption in FSC industries to achieve operational excellence. This study shows that with the adoption of blockchain FSC sectors can achieve sustainability in their operations and supply chain.
Deep Multi-Scale Representation Learning with Attention for Automatic Modulation Classification

Currently, deep learning methods with stacking small size convolutional filters are widely used for automatic modulation classification (AMC). In this report, we find some experienced improvements by using large kernel size for convolutional deep convolution neural network based AMC, which is more efficient in extracting multi-scale features of the raw signal I/Q sequence data. Also, Squeeze-and-Excitation (SE) mechanisms can significantly help AMC networks to focus on the more important features of the signal. As a result, we propose a multi-scale feature network with large kernel size and SE mechanism (SE-MSFN) in this paper. SE-MSFN achieves state-of-the-art classification performance on the public well-known RADIOML 2018.01A dataset, with average classification accuracy of 64.50%, surpassing CLDNN by 1.42%, maximum classification accuracy of 98.5%, and an average classification accuracy of 85.53% in the lower SNR range OdB to 10dB, surpassing CLDNN by 2.85%. In addition, we also verified that ensemble learning can help further improve classification performance. We hope this report can provide some references for developers and researchers in practical scenes.
Efficient Deep Visual and Inertial Odometry with Adaptive Visual Modality Selection

In recent years, deep learning-based approaches for visual-inertial odometry (VIO) have shown remarkable performance outperforming traditional geometric methods. Yet, all existing methods use both the visual and inertial measurements for every pose estimation incurring potential computational redundancy. While visual data processing is much more expensive than that for the inertial measurement unit (IMU), it may not always contribute to improving the pose estimation accuracy. In this paper, we propose an adaptive deep-learning based VIO method that reduces computational redundancy by opportunistically disabling the visual modality. Specifically, we train a policy network that learns to deactivate the visual feature extractor on the fly based on the current motion state and IMU readings. A Gumbel-Softmax trick is adopted to train the policy network to make the decision process differentiable for end-to-end system training. The learned strategy is interpretable, and it shows scenario-dependent decision patterns for adaptive complexity reduction. Experiment results show that our method achieves a similar or even better performance than the full-modality baseline with up to $$78.8\%$$ computational complexity reduction for KITTI dataset evaluation. The code is available at https://github.com/mingyuyng/Visual-Selective-VIO .
RIVIE: Robust Inherent Video Information Embedding

Imagine an interesting situation when watching a movie, we can scan the screen using our smartphones to get some extra information about this movie such as the cast, the release date, the movie's homepage, etc. Our prospect is a world where each video contains invisible information that can be delivered to us through mobile devices with cameras. This paper proposes the first deep learning-based information hiding method for videos to achieve information transmission from screens to cameras. Compared with hiding information in single images, the methods for videos need to maintain visual quality in both spatial and temporal domains. Furthermore, the training of video models builds on a large video dataset, which needs much more computational resources than training models for images. To reduce the computational complexity, we propose to simulate data on-the-fly to generate simulated sequences from single images. Then, we use the simulated data to train a spatio-temporal generator that hides information in videos while maintaining visual quality. During training, a temporal loss function based on the simulated data is exploited to ensure the temporal consistency of generated videos. After embedding, we use a decoder to recover the hidden information. To simulate the imaging pipeline from screens to cameras in the real world, we insert a distortion network between the generator and decoder. The distortion network is based on differentiable 3D rendering to cover possible distortions introduced in the procedure of camera imaging. Experimental results show that the hidden information in videos can be extracted by cameras without impacting the visual quality. Our work can be applied to many fields, such as advertisement, entertainment, and education.
Dunhuang Mural Line Drawing Based on Bi-Dexined Network and Adaptive Weight Learning

Dunhuang murals are an excellent cultural heritage, a masterpiece of Chinese painting, and a treasure of Buddhist art. A large number of murals and sculptures have gone through thousands of years and are of high artistic value. Digital line drawings of the Dun- huang Murals can not only show the beauty of the line art of murals, but also guide the restoration of murals. It belongs to the direction of image edge detection in computer vision. The purpose of edge detection is to quickly and accurately locate and extract image edge feature information. Although some traditional detection algorithms and methods based on deep learning have made some progress, they have not achieved ideal results in generating mural datasets. Compared to existing methods, we propose a novel edge detection architecture Bi-Dexined network. Firstly, the adaptive weight in this method can well balance the weight influence of the fusion of different levels of feature maps on the final prediction result. Secondly, The upsampling path can extract deeper semantic information in the network. After testing on several different edge detection and mural datasets, our method can generate clearer and more reasonable edge maps than other methods.
Revisiting Near/Remote Sensing with Geospatial Attention

This work addresses the task of overhead image segmentation when auxiliary ground-level images are available. Recent work has shown that performing joint inference over these two modalities, often called near/remote sensing, can yield significant accuracy improvements. Extending this line of work, we introduce the concept of geospatial attention, a geometry-aware attention mechanism that explicitly considers the geospatial relationship between the pixels in a ground-level image and a geographic location. We propose an approach for computing geospatial attention that incorporates geometric features and the appearance of the overhead and ground-level imagery. We introduce a novel architecture for near/remote sensing that is based on geospatial attention and demonstrate its use for five segmentation tasks. The results demonstrate that our method significantly outperforms the previous state-of-the-art methods.
Plausibility and Perception of Personalized Virtual Humans between Virtual and Augmented Reality

This article investigates the effects of different XR displays on the perception and plausibility of personalized virtual humans. We compared immersive virtual reality (VR), video see-through augmented reality (VST AR), and optical see-through AR (OST AR). The personalized virtual alter egos were generated by state-of-the-art photogrammetry methods. 42 participants were repeatedly exposed to animated versions of their 3D-reconstructed virtual alter egos in each of the three XR display conditions. The reconstructed virtual alter egos were additionally modified in body weight for each repetition. We show that the display types lead to different degrees of incongruence between the renderings of the virtual humans and the presentation of the respective environmental backgrounds, leading to significant effects of perceived mismatches as part of a plausibility measurement. The device-related effects were further partly confirmed by subjective misestimations of the modified body weight and the measured spatial presence. Here, the exceedingly incongruent OST AR condition leads to the significantly highest weight misestimations as well as to the lowest perceived spatial presence. However, similar effects could not be confirmed for the affective appraisal (i.e., humanness, eeriness, or attractiveness) of the virtual humans, giving rise to the assumption that these factors might be unrelated to each other.
IntentVizor: Towards Generic Query Guided Interactive Video Summarization

The target of automatic video summarization is to create a short skim of the original long video while preserving the major content/events. There is a growing interest in the integration of user queries into video summarization or query-driven video summarization. This video summarization method predicts a concise synopsis of the original video based on the user query, which is commonly represented by the input text. However, two inherent problems exist in this query-driven way. First, the text query might not be enough to describe the exact and diverse needs of the user. Second, the user cannot edit once the summaries are produced, while we assume the needs of the user should be subtle and need to be adjusted interactively. To solve these two problems, we propose IntentVizor, an interactive video summarization framework guided by generic multi-modality queries. The input query that describes the user's needs are not limited to text but also the video snippets. We further represent these multi-modality finer-grained queries as user ‘intent’, which is interpretable, interactable, editable, and can better quantify the user's needs. In this paper, we use a set of the proposed intents to represent the user query and design a new interactive visual analytic interface. Users can interactively control and adjust these mixed-initiative intents to obtain a more satisfying summary through the interface. Also, to improve the summarization quality via video understanding, a novel Granularity-Scalable Ego-Graph Convolutional Networks (GSE-GCN) is proposed. We conduct our experiments on two benchmark datasets. Comparisons with the state-of-the-art methods verify the effectiveness of the proposed framework. Code and dataset are available at https://github.com/jnzs1836/intentvizor.
Adaptive Agent Transformer for Few-Shot Segmentation

Few-shot segmentation (FSS) aims to segment objects in a given query image with only a few labelled support images. The limited support information makes it an extremely challenging task. Most previous best-performing methods adopt prototypical learning or affinity learning. Nevertheless, they either neglect to further utilize support pixels for facilitating segmentation and lose spatial information, or are not robust to noisy pixels and computationally expensive. In this work, we propose a novel end-to-end adaptive agent transformer (AAFormer) to integrate prototypical and affinity learning to exploit the complementarity between them via a transformer encoder-decoder architecture, including a representation encoder, an agent learning decoder and an agent matching decoder. The proposed AAFormer enjoys several merits. First, to learn agent tokens well without any explicit supervision, and to make agent tokens capable of dividing different objects into diverse parts in an adaptive manner, we customize the agent learning decoder according to the three characteristics of context awareness, spatial awareness and diversity. Second, the proposed agent matching decoder is responsible for decomposing the direct pixel-level matching matrix into two more computationally-friendly matrices to suppress the noisy pixels. Extensive experimental results on two standard benchmarks demonstrate that our AAFormer performs favorably against state-of-the-art FSS methods.
OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning

Semi-supervised learning (SSL) is one of the dominant approaches to address the annotation bottleneck of supervised learning. Recent SSL methods can effectively leverage a large repository of unlabeled data to improve performance while relying on a small set of labeled data. One common assumption in most SSL methods is that the labeled and unlabeled data are from the same data distribution. However, this is hardly the case in many real-world scenarios, which limits their applicability. In this work, instead, we attempt to solve the challenging open-world SSL problem that does not make such an assumption. In the open-world SSL problem, the objective is to recognize samples of known classes, and simultaneously detect and cluster samples belonging to novel classes present in unlabeled data. This work introduces OpenLDN that utilizes a pairwise similarity loss to discover novel classes. Using a bi-level optimization rule this pairwise similarity loss exploits the information available in the labeled set to implicitly cluster novel class samples, while simultaneously recognizing samples from known classes. After discovering novel classes, OpenLDN transforms the open-world SSL problem into a standard SSL problem to achieve additional performance gains using existing SSL methods. Our extensive experiments demonstrate that OpenLDN outperforms the current state-of-the-art methods on multiple popular classification benchmarks while providing a better accuracy/training time trade-off. Code: https://github.com/nayeemrizve/OpenLDN .
DeciWatch: A Simple Baseline for $$10\times $$ Efficient 2D and 3D Pose Estimation

This paper proposes a simple baseline framework for video-based 2D/3D human pose estimation that can achieve $$10\times $$ efficiency improvement over existing works without any performance degradation, named DeciWatch . Unlike current solutions that estimate each frame in a video, DeciWatch introduces a simple yet effective sample-denoise-recover framework that only watches sparsely sampled frames, taking advantage of the continuity of human motions and the lightweight pose representation. Specifically, DeciWatch uniformly samples less than $$10\%$$ video frames for detailed estimation, denoises the estimated 2D/3D poses with an efficient Transformer architecture, and then accurately recovers the rest of the frames using another Transformer-based network. Comprehensive experimental results on three video-based human pose estimation, body mesh recovery tasks and efficient labeling in videos with four datasets validate the efficiency and effectiveness of DeciWatch. Code is available at https://github.com/cure-lab/DeciWatch .
Motion-Driven Spatial and Temporal Adaptive High-Resolution Graph Convolutional Networks for Skeleton-Based Action Recognition

Graph convolutional networks (GCN) have attracted increasing interest in action recognition in recent years.GCN models human skeleton sequences as spatio-temporal graphs.Also, attention mechanisms are often jointly used with GCNs to highlight important frames or body joints in a sequence.However, attention modules learn parameters offline and are fixed, so may not adapt well to various action samples.In this paper, we propose a simple but effective motion-driven spatial and temporal adaptation strategy to dynamically strengthen the features of important frames and joints for skeleton-based action recognition.The rationale is that the joints and frames with dramatic motions are generally more informative and discriminative.We decouple and combine the spatial and temporal refinements by using a two-branch structure, in which the joint and frame-wise feature refinements perform in parallel.Such a structure can also lead to learn more complementary feature representations.Moreover, we propose to use the fully connected graph convolution to learn the long-range spatial dependencies.Besides, we investigate two high-resolution skeleton graphs by creating virtual joints, aiming to improve the representation of skeleton features.By combining the above proposals, we develop a novel motion-driven spatial and temporal adaptive high-resolution GCN.Experimental results demonstrate that the proposed model achieves state-of-the-art (SOTA) results on the challenging large-scale Kinetics-Skeleton and UAV-Human datasets, and it is on par with the SOTA methods on the two NTU-RGB+D 60&120 datasets.Additionally, our motiondriven adaptation method shows encouraging performance when compared with the attention mechanisms.
Bring Evanescent Representations to Life in Lifelong Class Incremental Learning

In Class Incremental Learning (CIL), a classification model is progressively trained at each incremental step on an evolving dataset of new classes, while at the same time, it is required to preserve knowledge of all the classes ob-served so far. Prototypical representations can be lever-aged to model feature distribution for the past data and in-ject information of former classes in later incremental steps without resorting to stored exemplars. However, if not up-dated, those representations become increasingly outdated as the incremental learning progresses with new classes. To address the aforementioned problems, we propose a frame-work which aims to (i) model the semantic drift by learning the relationship between representations of past and novel classes among incremental steps, and (ii) estimate the feature drift, defined as the evolution of the represen-tations learned by models at each incremental step. Se-mantic and feature drifts are then jointly exploited to infer up-to-date representations of past classes (evanescent rep-resentations), and thereby infuse past knowledge into incre-mental training. We experimentally evaluate our framework achieving exemplar-free SotA results on multiple bench-marks. In the ablation study, we investigate nontrivial relationships between evanescent representations and models.
Improving Deep Learning Based Anomaly Detection on Multivariate Time Series Through Separated Anomaly Scoring

The importance of anomaly detection in multivariate time series has led to the development of several prominent deep learning solutions. As a part of the anomaly detection method, the scoring method has shown to be of significant importance when separating non-anomalous points from anomalous ones. At this time, most of the solutions utilize an aggregated score which means that relevant information created by the anomaly detection model might be lost. Therefore, this study has set out to examine to what extent anomaly detection in multivariate time series based on deep learning can be improved if all the residuals from each individual channel is considered in the anomaly score. To achieve this, an aggregated and separated scoring method has been applied with a simple denoising convulutional autoencoder (DCAE). In addition, the performance has been compared with other state-of-the-art methods. The result showed that the separated approach has the potential to generate a significantly higher performance than the aggregated one. At the same time, there were some indications suggesting that an aggregated scoring is better at generalizing when no labels to base the anomaly thresholds on, are available. Therefore, the result should serve as an encouragement to use a separated scoring approach together with a small sample of labeled anomalies to optimise the thresholds. Lastly, due to the impact of the anomaly score, the result suggests that future research within this field should consider applying the same anomaly scoring method when comparing the performance of deep learning algorithms.
Temporal and Cross-modal Attention for Audio-Visual Zero-Shot Learning

Audio-visual generalised zero-shot learning for video classification requires understanding the relations between the audio and visual information in order to be able to recognise samples from novel, previously unseen classes at test time. The natural semantic and temporal alignment between audio and visual data in video data can be exploited to learn powerful representations that generalise to unseen classes at test time. We propose a multi-modal and Temporal Cross-attention Framework (TCaF) for audio-visual generalised zero-shot learning. Its inputs are temporally aligned audio and visual features that are obtained from pre-trained networks. Encouraging the framework to focus on cross-modal correspondence across time instead of self-attention within the modalities boosts the performance significantly. We show that our proposed framework that ingests temporal features yields state-of-the-art performance on the UCF-GZSL $$^{cls}$$ , VGGSound-GZSL $$^{cls}$$ , and ActivityNet-GZSL $$^{cls}$$ benchmarks for (generalised) zero-shot learning. Code for reproducing all results is available at https://github.com/ExplainableML/TCAF-GZSL .