Reviewerly/exHarmony huggingface dataset:
Online Learning of Neural Surface Light Fields Alongside Real-Time Incremental 3D Reconstruction

Immersive novel view generation is an important technology in the field of graphics and has recently also received attention for operator-based human-robot interaction. However, the involved training is time-consuming, and thus the current test scope is majorly on object capturing. This limits the usage of related models in the robotics community for 3D reconstruction since robots (1) usually only capture a very small range of view directions to surfaces that cause arbitrary predictions on unseen, novel direction, (2) requires real-time algorithms, and (3) work with growing scenes, e.g., in robotic exploration. The letter proposes a novel Neural Surface Light Fields model that copes with the small range of view directions while producing a good result in unseen directions. Exploiting recent encoding techniques, the training of our model is highly efficient. In addition, we design Multiple Asynchronous Neural Agents (MANA), a universal framework to learn each small region in parallel for large-scale growing scenes. Our model learns online the Neural Surface Light Fields (NSLF) aside from real-time 3D reconstruction with a sequential data stream as the shared input. In addition to online training, our model also provides real-time rendering after completing the data stream for visualization. We implement experiments using well-known RGBD indoor datasets, showing the high flexibility to embed our model into real-time 3D reconstruction and demonstrating high-fidelity view synthesis for these scenes.
Can I Own Your NFTs? Understanding the New Attack Surface to NFTs

Recent years have witnessed the increasing popularity and market value of Non-Fungible Tokens (NFTs), along with the burgeoning of blockchains and metaverse. The media hypes often imply that NFTs are as secure as the underlying blockchains. In this work, we take a first look into the building blocks of NFTs (i.e., ownership certificates stored on-chain and the metadata and digital assets stored on-chain or off-chain), focusing on understanding the new attack surface and safety of these digital assets (rather than the traditional attacks on block-chains). For this purpose, we provide a detailed analysis of the logical structure of the dominant off-chain NFTs, followed by the attack surface analysis. Our study indicates that specific new attacks could be easily mounted on them. To validate our findings, we experiment by minting our own NFTs on Ethereum and demonstrate that we can successfully mount the attacks with trivial or even no cost. The cause of these new attacks is rooted in the current design of NFTs where the digital assets are decoupled from the contracts deployed on the blockchains. We discuss some future research to address these vulnerabilities.
Silent Speech Interface With Vocal Speaker Assistance Based on Convolution-Augmented Transformer

Silent Speech Interface (SSI) based on neuromuscular signals has become popular in Human-Computer Interaction. Articulatory neuromuscular activity can be captured with surface electromyography (sEMG). However, in the absence of acoustic sounds, it remains challenging to convert the articulatory neuromuscular signals of silent speakers into corresponding audio. This paper proposes a Mandarin-based SSI, a novel converter between articulatory neuromuscular from silent speakers and audio signals, by providing an auxiliary set of sEMG and audio pairs from vocal speakers. The proposed method is based on a hybrid framework combining convolutional neural networks and Transformer to utilize both global and local information. We experimentally validate the feasibility of the proposed method by obtaining an average objective character error rate (CER) of 10.69% using an automatic speech recognition (ASR) evaluation tool with four silent speakers. The results show that our Mandarin-based SSI with vocal speaker assistance facilitates the conversion from sEMG into audio in the cross-subject scenario.
Learning End-to-End Visual Servoing Using an Improved Soft Actor-Critic Approach With Centralized Novelty Measurement

End-to-end visual servoing based on reinforcement learning(RL) can simplify the design of features and control laws and has strong scalability in combination with neural networks. However, it is challenging for RL-based VS tasks to operate in a continuous state or action space due to the difficulty in space exploration and slow training convergence. Hence, this paper presents a novel measurement method based on centralized features extracted by a neural network, which calculates the novelty of the visited state to encourage RL-agent exploration. Moreover, we propose a hybrid probability sampling method that improves the Prioritized Experience Replay (PER) based on a Temporal-Difference (TD) error by integrating the intrinsic and external rewards. This strategy represents the transition novelty and quality in the buffer replay, respectively, to promote convergence in the training process. Finally, we develop an end-to-end VS scheme based on the maximum entropy RL Soft Actor-Critic (SAC). Several simulated experiments in CoppeliaSim are designed for end-to-end VS, where the target detection information is the agent’s input. The results highlight that our method’s reward value and completion rates are 0.35 and 8.0% higher than the SAC VS baseline. At the same time, we conduct experiments to verify the effectiveness of the proposed algorithm.
MisMatch: Calibrated Segmentation via Consistency on Differential Morphological Feature Perturbations With Limited Labels

Semi-supervised learning (SSL) is a promising machine learning paradigm to address the ubiquitous issue of label scarcity in medical imaging. The state-of-the-art SSL methods in image classification utilise consistency regularisation to learn unlabelled predictions which are invariant to input level perturbations. However, image level perturbations violate the cluster assumption in the setting of segmentation. Moreover, existing image level perturbations are hand-crafted which could be sub-optimal. In this paper, we propose MisMatch, a semi-supervised segmentation framework based on the consistency between paired predictions which are derived from two differently learnt morphological feature perturbations. MisMatch consists of an encoder and two decoders. One decoder learns positive attention for foreground on unlabelled data thereby generating dilated features of foreground. The other decoder learns negative attention for foreground on the same unlabelled data thereby generating eroded features of foreground. We normalise the paired predictions of the decoders, along the batch dimension. A consistency regularisation is then applied between the normalised paired predictions of the decoders. We evaluate MisMatch on four different tasks. Firstly, we develop a 2D U-net based MisMatch framework and perform extensive cross-validation on a CT-based pulmonary vessel segmentation task and show that MisMatch statistically outperforms state-of-the-art semi-supervised methods. Secondly, we show that 2D MisMatch outperforms state-of-the-art methods on an MRI-based brain tumour segmentation task. We then further confirm that 3D V-net based MisMatch outperforms its 3D counterpart based on consistency regularisation with input level perturbations, on two different tasks including, left atrium segmentation from 3D CT images and whole brain tumour segmentation from 3D MRI images. Lastly, we find that the performance improvement of MisMatch over the baseline might originate from its better calibration. This also implies that our proposed AI system makes safer decisions than the previous methods.
Oriented Infrared Vehicle Detection in Aerial Images via Mining Frequency and Semantic Information

Infrared vehicle detection based on aerial images has significant applications in military and civilian fields for the perception ability under low-light and foggy conditions. However, it remains challenging due to the following characteristics. First, infrared textures and edges are blurred, implying a plenty of low-frequency signals and a shortage of detailed descriptions. Second, objects in infrared images present different patterns depending on their thermal radiations, which hamper the feature extraction of convolution kernels. Third, infrared images lack color information, which means fewer features for classification and regression can be used. Inspired by cognitive neuroscience that humans perceive the entirety from low-frequency information and discern details from high-frequency information, we devise a new framework for oriented infrared vehicle detection called I <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> MDet (Infrared Information Mining Detector) to tackle the above challenges. It consists of two significant designs: the kaleidoscope module and the semantic feature supplement module (SFSM). In the kaleidoscope module, we explore the effect of kernel sizes and dilation rates on frequency information mining with kaleidoscope-like equivalent kernels. Features in this module are extracted by adaptive involution operators instead of convolution kernels to deal with multiple patterns. The SFSM provides the network with features beneficial for classification and regression. On the one hand, the network is guided to learn more meaningful features under semantic supervision. On the other hand, features output by the SFSM supplement the detection head with semantic information. Experimental results on the public dataset DroneVehicle demonstrate that our proposed approach achieves outstanding performance on oriented infrared vehicle detection.
DRT: Deformable Region-Based Transformer for Nonrigid Medical Image Registration With a Constraint of Orientation

Nonrigid medical image registration is a key technology for clinical disease diagnosis by integrating the vital information taken from input images acquired using multiple imaging sensors. Recently, transformer networks have attracted attention in image registration, but splitting images into patches remains a problem. Although there are usually multiple structural regions with different sizes and shapes in medical images, current transformer methods usually use an equal-sized or rectangular patch embedding that cannot represent the structural differences and directionality of different regions. In addition, it may also destroy the semantic consistency of objects. To address this problem, we propose a new deformable region-based structural relevance embedding (DRSRE) module which learns to adaptively split the images into deformable structural regions of different sizes and shapes with orientation constraints rather than using equal-sized or rectangular patches. In this way, our method can represent the structural differences and directionality between regions and effectively preserve the semantic consistency in regions to calculate accurate transformation relationships for nonrigid image registration. We term the DRSRE module with a transformer part as a deformable region-based transformer (DRT) and conduct extensive evaluations of DRT in the context of nonrigid medical image registration, and the results validate that the proposed DRT outperforms the state-of-the-art nonrigid registration methods, achieving 0.748 and 0.886 Dice scores with 0.217% and 0.308% foldings for deformation field on the LPBA40 and ADNI datasets.
Learned Lossless Image Compression Through Interpolation With Low Complexity

With the increasing popularity of deep learning in image processing, many learned lossless image compression methods have been proposed recently. One group of algorithms are based on scale-based auto-regressive models and can provide competitive compression performance while also allowing easily parallelized computations and short encoding/decoding times. However, they use large neural networks and have high computational requirements. This paper presents an interpolation based learned lossless image compression method which falls in the scale-based auto-regressive models group. The method achieves compression performance better than or on par with the recent scale-based auto-regressive models, yet requires more than 10x less neural network parameters (0.19M) and encoding/decoding computation complexity. These achievements are due to the contributions/findings in the overall system and neural network architecture design, such as sharing interpolator neural networks across different scales, using separate neural networks for different parameters of the probability distribution model and performing the processing in the YCoCg-R color space instead of the RGB color space.
Learning to Infer Unseen Single-/ Multi-Attribute-Object Compositions With Graph Networks

Inferring the unseen attribute-object composition is critical to make machines learn to decompose and compose complex concepts like people. Most existing methods are limited to the composition recognition of single-attribute-object, and can hardly learn relations between the attributes and objects. In this paper, we propose an attribute-object semantic association graph model to learn the complex relations and enable knowledge transfer between primitives. With nodes representing attributes and objects, the graph can be constructed flexibly, which realizes both single- and multi-attribute-object composition recognition. In order to reduce mis-classifications of similar compositions (e.g., scratched screen and broken screen), driven by the contrastive loss, the anchor image feature is pulled closer to the corresponding label feature and pushed away from other negative label features. Specifically, a novel balance loss is proposed to alleviate the domain bias, where a model prefers to predict seen compositions. In addition, we build a large-scale Multi-Attribute Dataset (MAD) with 116,099 images and 8,030 label categories for inferring unseen multi-attribute-object compositions. Along with MAD, we propose two novel metrics Hard and Soft to give a comprehensive evaluation in the multi-attribute setting. Experiments on MAD and two other single-attribute-object benchmarks (MIT-States and UT-Zappos50K) demonstrate the effectiveness of our approach.
Towards Accurate Image Matching by Exploring Redundancy Between Multiple Descriptors

Finding correspondences between a pair of images is the key ingredient for many applications such as localization and panorama. However, due to a variety of challenges between multi-view images in practice, the results of using a single kind of descriptor may vary significantly across different scenes. In this paper, we treat the assignment task as a clustering problem and propose an image matching method that fuses multiple descriptors to tackle the above difficulties. First, we extract multiple descriptors at the keypoints on two images. Then, we compute a pairwise similarity matrix for each kind of descriptor. Afterwards, we compute a weighted combination of these similarity matrices, and use it to build correspondences via a modified multi-kernel clustering module. The proposed method is tested on three public image datasets: two ground image sets and an Unmanned Aerial Vehicle (UAV) image set. Experiments show that the proposed method can adapt to different number of descriptors. It significantly improves the matching accuracy in a variety of scenarios and downstream tasks.
MF-GCN: Motion Flow-Based Graph Network Learning Dynamics for Aerial IR Target Recognition

The primary technology of the infrared imaging seeker optoelectronic countermeasure system includes the automatic recognition and tracking of air targets. New complex infrared interference factors pose severe challenges to the ability of infrared thermal imaging seekers to identify and lock targets accurately. Despite significant recent advances, machine vision systems still lag substantially behind biological vision systems in terms of performance and robustness. The distinguishing feature of the brain is its ability to detect learned objects under various non-ideal situations. Inspired by the sensitivity of the dorsal visual pathway to object motion information, a graph convolutional network learning model is developed for aerial target recognition, called the adaptive graph reasoning method driven by motion flow model, which employs the predicted object optical flow information to promote the keypoint estimation performance. First, an adaptive graph fusion module is designed to establish an adaptation mechanism to generalize keypoint features from the regular network to the target skeleton graph model. Then, the optical flow estimation features are utilized to understand the graph reasoning of the target keypoint motion. Finally, we propose a aerial target pose estimation framework which effectively integrates the temporal cues across frames by combining the motion features of optical flow estimation, and effectively promotes the estimation accuracy of keypoints. Finally, our extensive experiments in the established aerial target flight infrared dataset show the efficiency of the presented model.
Comparative Study of the Effect of Various Color Models on Classification of Parasitic Eggs

Parasitic infections are now recognized by the World Health Organization (WHO) as one of the main causes of illness. Automating routine feces inspection for parasitic illnesses is the goal of this challenge, which aims to bring together experts in the field to develop reliable automated algorithms to detect and categorize parasitic worm eggs in a range of microscopic images. In this study, we evaluated how various color models affected the classification of 11 different types of parasite eggs from fecal smear samples. The input of the classification algorithm is the preprocessed microscopic image. Seven color spaces classifiers have been tested on the Inception-ResN et-V2 CNN model. Additionally, an analysis of variance (ANOVA) has been conducted to compare the expected error rate of those seven classifiers. The models were tested on 2,200 entirely separated images after being trained on 11,000 images. Out of seven color spaces on the test dataset, the XYZ color space performed better (accuracy: 82.77%, recall: 82.77%, precision: 84.53%, and Fl:83.64%). According to the results of the ANOVA analysis, the error rates produced by those seven classifiers are not statistically different.
E-Payment System to Reduce Use of Paper Money for Daily Transactions

This paper presents a framework of an E-payment system which can be used for daily petty cash transactions. The framework involves communication of encrypted messages through an insecure network using Advanced Encryption System (AES) with a timestamp key. To ensure the message authentication and integrity, the Hash-based Message Authentication Code (HMAC) is used. The associated key in HMAC is considered as a function of the phone Media Access Control (MAC) address as well as the user's biometric data for additional security. Timestamp key is updated each time a transaction takes place. Thus, the framework can be implemented to replace the use of paper money for daily transactions.
Improving Performance Factors of an Imbalanced Credit Risk Dataset Using SMOTE

In this paper, a supervised machine learning approach for predicting the credit risk of a loan applicant has been presented. Due to the imbalance of real-world datasets, often high accuracy but low precision, recall, and F1-scores is obtained. This paper aims to solve these issues by using oversampling technique called Synthetic Minority Over-sampling Technique (SMOTE). Lending Club, a real-world imbalanced dataset has been considered to train the models. The dataset consists of 151 features among which 113 columns are numerical and the rest are categorical. It had a total of 2260701 instances. SMOTE is applied to the dataset to make it balanced improving accuracy, precision, recall, and F1-score. Four supervised classifier algorithms such as Logistic Regression, XGBoost, Decision Tree, and Random Forest were used to predict the loan defaulters. Among them, Random Forest achieves the highest accuracy of 87% with precision, recall, and F1-score of 0.81, 0.97, and 0.89 respectively. Also, the ROC-AVC score of Random Forest was 0.93.
Disguised Face Recognition using Convolutional Neural Network

In our day-to-day life, we are implementing various biometric technologies especially facial recognition to ensure privacy of our information as well as for secured access. But facial recognition has become a tough task recently as faces could be covered with various disguises. In this paper, disguised face recognition has been taken into consideration by utilizing Convolutional Neural Network (CNN) approach. CNN architectures have a lot of variations optimized for various purposes. FaceNet is one of them and performed really well on facial image dataset. The main purpose of this work is to recognize the human faces covered with various disguises like sunglass, hat, beard, mustache, hair, scarf etc. Multi Task Cascaded Convolutional Network (MTCNN) has been taken into account to detect facial regions and FaceNet architecture has been utilized to find facial embeddings. After that, Support Vector Machine (SVM) has been used to categorize the similarity of faces. Our model achieved an accuracy of 88.78% on IIIT-Delhi Disguise Version 1 Face Dataset and the result has been compared with the state-of-the-art architecture. It has been shown that our model surpassed the state-of-the-art architecture.
Real-Time and Accurate Monitoring of Social Distance and Contact Tracing during COVID-19 Situation

In recent years, a lot of research works have been done on object detection using various machine learning models. However, not many works have been done on detecting and tracking humans in particular. This study works with the YOLOv4 object detector to detect humans to use the detections for maintaining social distance. For this study, the YOLOv4 model is trained on only one class named “Person”. This is done to improve the speed of detecting humans in real time scenario with satisfying accuracy of 97% to 99%. These detections are then tracked to build a system for maintaining social distance and alerting the authority if a breach in the social distance is detected. This system can be applied at ticket counters, hospitals, offices, factories etc. It can also be used for maintaining social distance among the students and the teachers in the classroom for their safety.
Bangla Language Dialect Classification using Machine Learning

Dialect classification of a language is a complex work as it is the variation of the same language. This paper classifies dialect based on local Bengali text. The classification becomes harder when it is about a language that is not very much available in written format or stored in any other way except spoken among local people. For natural language processing (NLP) a good amount of data is essential to get the job done. It focuses on generating an enriched dataset of the local Bangla language. The dataset introduces two popular dialects which are Chatgaiya and Pabna which are spoken by a large number of people. It comprises about 5000 data regarding these local languages which are annotated with their respective dialects. A five-step Exploratory Data Analysis (EDA) is carried out. Feature extraction is conducted using three different techniques like CountVectorizer, Term Frequency-Inverse Document Frequency (TF-IDF) and Word2vec. With this huge amount of data, it worked on classifying Bangla language dialect using machine learning algorithms such as Support Vector Machine (SVM), Multinomial Naïve Bayes (MNB), Logistic Regression (LR), Random Forest (RF), Decision Tree (DT), K Nearest Neighbor (KNN). This study obtained the highest 96% accuracy.