Reviewerly/exHarmony huggingface dataset:
Effective Ambiguity Attack Against Passport-based DNN Intellectual Property Protection Schemes through Fully Connected Layer Substitution

Since training a deep neural network (DNN) is costly, the well-trained deep models can be regarded as valuable intellectual property (IP) assets. The IP protection associated with deep models has been receiving increasing attentions in recent years. Passport-based method, which replaces normalization layers with passport layers, has been one of the few protection solutions that are claimed to be secure against advanced attacks. In this work, we tackle the issue of evaluating the security of passport-based IP protection methods. We propose a novel and effective ambiguity attack against passport-based method, capable of successfully forging multiple valid passports with a small training dataset. This is accomplished by inserting a specially designed accessory block ahead of the passport parameters. Using less than 10% of training data, with the forged passport, the model exhibits almost indistinguishable performance difference (less than 2%) compared with that of the authorized passport. In addition, it is shown that our attack strategy can be readily generalized to attack other IP protection methods based on watermark embedding. Directions for potential remedy solutions are also given.
SATA M.2 on Forensics: Trim Function Effect on Recovering Permanently Deleted Files

The spread of different types of SSD memory in a wide range of devices increases the challenge of cybercrime and forensic investigation. This is due to the features of the memory structure and how the data is recovered under such features. This research paper consists of an experiment to recover permanently deleted files in SATAM.2 SSD memory when- the Trim function is disabled and permitted with various forensic tools such as OSForensics, Autopsy, FTK and AXIOM. The experiment is applied to the NTFS file system under the Windows 11 environment. The research finds that 0% of files are recovered when the Trim function is enabled and 100% of files can be recovered if Trim function is disabled. This makes the recovery process difficult for investigators to find valid evidence. In future work, there is a need for a method that allows investigators to recover files in such conditions, and to apply more experiments to various features under different file system types and operating systems.
A Hierarchical Path Planner Based on Even-order Hyperelliptic Orbital Limit-Cycles

This paper proposes a Hierarchical Path Planner (HPP) based on Even-order Hyperelliptic Orbital Limit-Cycles (EHOLC). Traditional Elliptic Orbital Limit-Cycles (EOLC) have been widely studied. However, in structured rectangular obstacle environments, ellipses have a higher porosity when enveloping rectangles, resulting in lower environmental adaptability. Therefore, EHOLC is introduced, and the hyperelliptic equation order of obstacles is determined based on a set porosity. Then, the differential equation of the EHOLC is established, and the trajectory is solved. Finally, a hierarchical path planner is proposed, which selects obstacle avoidance controllers and target reaching controllers according to the environment map to solve the entire path. Simulation experiments show that compared with traditional EOLC, the EHOLC-based HPP has higher map space utilization, shorter path planning length, and maintains flexibility, smoothness, and safety.
Differential Enhancement and Commonality Fusion for RGBT Tracking

Target tracking is a fundamental task and an important part in the field of computer vision. The image information contained in the visible (RGB) modality and thermal infrared (T) modality has differences. RGBT tracking utilizes this difference to fuse the information of the two modalities, fully combining the uniqueness and complementarity of the information of the two modalities. How to maximize the uniqueness of individual modalities in RGBT tracking and how to effectively combine the complementarity of the two modalities is the key issue in RGBT tracking. For the combination of the uniqueness and complementarity of the two modalities, a novel Differential Enhancement and Commonality Fusion Network (DECFNet) is designed in this paper, in which the main components include a feature enhancement module for the visible modality and a feature enhancement module for the thermal infrared modality, as well as a fusion module that effectively fuses the information of both. The results on three RGB T tracking benchmark datasets demonstrate the effectiveness of the proposed method.
MAN: Mix-domain attentional single-stage object detection network with multiple attention mechanisms

Deep convolutional neural networks (CNN) have achieved remarkable progress in the image processing field during the past decade. However, most existing CNN-based object detection algorithms mainly delve into network-deepening, overlooking the exploration of the inherent relationship amongst the features. To address this issue, an end-to-end Mix-domain Attention Network (MAN) is proposed. First, the backbone and head of MAN adopt the structure of YOLOv5 (YOLOv5 is a family of object detection architectures and models pre-trained on the COCO dataset by Ultralytics). Then, we embed two key modules in the backbone and head: non-local blocks and stereoscopic attention group (SAG). Non-local blocks are different from CNN’s local operations in that they can capture long-distance spatial contextual information (Such as the pixel-to-pixel relationship in images, and the pixel-to-pixel relationship between different frames in video streams) and combined with the residual structure, this allows the feature dimensions of the input and output to be consistent. Subsequently, the features extracted from the non-local blocks will be further passed through a trainable stereoscopic attention group to calculate the correlation of channel-to-channel, pixel-to-pixel, and update the output features. The Non-local operation is the self-attention mechanism and the SAG operation is the channel and spatial attention mechanism. Our innovation is the combination of these two operations to form a Mix-domain attention mechanism. The Non-local increases the connection between the pixel points. The SAG allows the model to focus more on the places that need attention. The combination of these two modules enhances recognition accuracy. The experimental results demonstrate that the two versions of a Mixdomain Attention Network achieves the following results on Microsoft Common Objects in Context (MS-COCO) with an input size of 640*640: AP(Average Precision)=38.2%, 44.4%,AP05 (IOU >0.5 Average Precision)=59.3%,63.4%, and higher than some mainstream target recognition models.
Underwater Image Enhancement Algorithm Based on Transformer and Convolutional Network

Underwater images suffer from color distortion, key information loss and blurred details, and it is difficult to reconstruct high-quality images. Therefore, we propose a U-shaped codec network that integrates Transformer and convolution for underwater image enhancement. First, we scale the degradation image to the size of 256$\times$ 256$\times$ 3 and use it as input. Next, we propose the improved encoder and decoder, the feature extraction encoding module that mixes local-ViT and global-ViT is constructed and the convolutional decoding module incorporating channel attention is designed. Finally, extensive experiments reveal that our algorithm can effectively achieve color recovery and detail enhancement of underwater images.
3D Hand Pose Estimation and Shape Reconstruction based on Multi-task Learning

3D hand pose estimation and shape reconstruction is an important problem in the fields of computer vision and machine learning. It aims to infer the 3D pose and shape of a hand from a given hand image, and has wide applications such as virtual reality, gesture recognition, medical image processing, among others. However, due to the limitations of current hardware computing resources, hand reconstruction algorithms must achieve a balance between speed and accuracy. To address this, we propose a real-time 3D hand shape reconstruction algorithm from a single RGB image. In order to reduce the parameters of the algorithm and improve the inference speed, we use a multitask learning approach to simultaneously predict the 2D and 3D coordinates of the hand. The predicted 3D coordinates are constrained by weak perspective projection to project onto 2D coordinates, and then a multi-stage refinement method is proposed to optimize the predicted 3D coordinates. Finally, testtime augmentation is added to further improve the model’s accuracy. Our proposed network achieves the fastest inference speed with the least parameters, and achieves state-of-the-art performance on the public hand dataset FreiHAND.
Hierarchical Spatio-Temporal Neural Network with Displacement Based Refinement for Monocular Head Pose Prediction

Head pose prediction aims to forecast future head pose given observed sequence, which plays an increasingly important role in human computer interaction, virtual reality, and driver monitoring. However, since there are many moving possibilities, current head pose works, mainly focusing on estimation, fail to provide sufficient temporal information to meet the high demands for accurate predictions. This paper proposes (A) a Spatio-Temporal Encoder (STE), (B) a displacement based offset generating module, and (C) a time step feature aggregation module. The STE extracts spatial information via Transformer and temporal information according to the time order of frames. The displacement based offset generating module utilizes displacement information through a frequency domain process between adjacent frames to generate an offset to refine the prediction result. Furthermore, the time step feature aggregation module integrates time step features based on the information density and hierarchically extracts past motion information as prior knowledge to capture the motion recurrence. Extensive experiments have shown that the proposed network outperforms related methods, achieving a Mean Absolute Error (MAE) of 4.5865° on simple background sequences and 7.1325° on complex background sequences.
Mixed Distillation for Unsupervised Anomaly Detection

Anomaly detection is typically a class of unsupervised learning problems in which the model is trained with only normal samples. Knowledge distillation (KD) has shown promising results in the field of image anomaly detection, especially for texture images. However, the knowledge of the classical KD model is step-by-step transferred from the shallow layers to the deep, which causes the deep layers not to be well-fitted due to an incomplete match of the shallow layers of the student network. For this problem, we propose a skip distillation method, which allows the deep layers of the student network to learn directly from the shallow of the teacher, avoiding a worse deep fit. We also design a symmetric path that allows the shallow layers of the student network to learn directly from the deep of the teacher. These two paths encode suﬃcient information for the student network. We have done thorough experiments on the anomaly detection benchmark dataset MvtecAD, and the experimental results show that our model exceeds the current state-of-the-art anomaly detection methods in terms of texture classes.
Tackling Face Verification Edge Cases: In-Depth Analysis and Human-Machine Fusion Approach

Nowadays, face recognition systems surpass human performance on several datasets. However, there are still edge cases that the machine can’t correctly classify. This paper investigates the effect of a combination of machine and human operators in the face verification task. First, we look closer at the edge cases for several state-of-the-art models to discover common datasets’ challenging settings. Then, we conduct a study with 60 participants on these selected tasks with humans and provide an extensive analysis. Finally, we demonstrate that combining machine and human decisions can further improve the performance of state-of-the-art face verification systems on various benchmark datasets. Code and data are publicly available on GitHub <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Human Pose Prediction by Progressive Generation in Multi-scale Frequency Domain

We address a problem of 3D human pose prediction from a sequence of human body skeletons. To model the spatio-temporal dynamics, the discrete cosine transform (DCT) and the graph convolutional networks (GCN) are often applied to signals on a human skeleton graph. By DCT, temporal information of a human skeleton sequence can be embedded into the frequency domain. However, in previous studies, the prediction models using DCT implicitly learned each frequency coeﬃcient by gradients calculated from a loss of the predictions and the ground truths of human body skeletons. In this paper, we propose a progressive human pose prediction model in frequency domain so that explicitly predict high-, medium-, and low-frequency motion of a target person. We confirmed that the proposed method improves prediction accuracy through experiments using public datasets on Human3.6M and CMU Mocap datasets.
Age Prediction From Face Images Via Contrastive Learning

This paper presents a novel approach for accurately estimating age from face images, which overcomes the challenge of collecting a large dataset of individuals with the same identity at different ages. Instead, we leverage readily available face datasets of different people at different ages and aim to extract age-related features using contrastive learning. Our method emphasizes these relevant features while suppressing identity-related features using a combination of cosine similarity and triplet margin losses. We demonstrate the effectiveness of our proposed approach by achieving state-of-the-art performance on two public datasets, FG-NET and MORPH II.
Image Impression Estimation by Clustering People with Similar Tastes

This paper proposes a method for estimating impressions from images according to the personal attributes of users so that they can find the desired images based on their tastes. Our previous work, which considered gender and age as personal attributes, showed promising results, but it also showed that users sharing these attributes do not necessarily share similar tastes. Therefore, other attributes should be considered to capture the personal tastes of each user well. However, taking more attributes into account leads to a problem in which insufficient amounts of data are served to classifiers due to the explosion of the number of combinations of attributes. To tackle this problem, we propose an aggregation-based method to condense training data for impression estimation while considering personal attribute information. For evaluation, a dataset of 4,000 carpet images annotated with 24 impression words was prepared. Experimental results showed that the use of combinations of personal attributes improved the accuracy of impression estimation, which indicates the effectiveness of the proposed approach.
Interpreting Art by Leveraging Pre-Trained Models

In many domains, so-called foundation models were recently proposed. These models are trained on immense amounts of data resulting in impressive performances on various downstream tasks and benchmarks. Later works focus on leveraging this pre-trained knowledge by combining these models. To reduce data and compute requirements, we utilize and combine foundation models in two ways. First, we use language and vision models to extract and generate a challenging language vision task in the form of artwork interpretation pairs. Second, we combine and fine-tune CLIP as well as GPT-2 to reduce compute requirements for training interpretation models. We perform a qualitative and quantitative analysis of our data and conclude that generating artwork leads to improvements in visual-text alignment and, therefore, to more proficient interpretation models <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . Our approach addresses how to leverage and combine pre-trained models to tackle tasks where existing data is scarce or difficult to obtain.
TinyPedSeg: A Tiny Pedestrian Segmentation Benchmark for Top-Down Drone Images

The usage of Unmanned Aerial Vehicles (UAVs) has significantly increased in various fields such as surveillance, agriculture, transportation, and military operations. However, the integration of UAVs in these applications requires the ability to navigate autonomously and detect/segment objects in real-time, which can be achieved through the use of neural networks. Despite object detection for RGB images/videos obtained from UAVs are widely studied before, limited effort has been made for segmentation from top-down aerial images. Considering the case in which the UAV is extremely high from the ground, the task can be formed as tiny object segmentation. Thus, inspired from the TinyPerson dataset which focuses on person detection from UAVs, we present TinyPedSeg, which contains 2563 pedestrians in 320 images. Specialized only in pedestrian segmentation, our dataset presents more informativeness than other UAV segmentation datasets. The dataset and the baseline codes are available at https://github.com/ituvisionlab/tinypedseg
Grid Sample Based Temporal Iteration and Compactness-coefficient Distance for High Frame and Ultra-low Delay SLIC Segmentation System

High frame rate and ultra-low delay vision systems, which process 1000 FPS videos within 1 ms/frame delay, play an increasingly important role in fields such as robotics and factory automation. Among them, an image segmentation system is necessary as segmentation is a crucial pre-processing step for various applications. Recently many existing researches focus on superpixel segmentation, but few of them attempt to reach high processing speed. To achieve this target, this paper proposes: (A) Grid sample based temporal iteration, which leverages the high frame rate video property to distribute iterations into the temporal domain, ensuring the entire system is within one frame delay. Additionally, grid sample is proposed to add initialization information to temporal iteration for the stability of superpixels. (B) Compactness-coefficient distance is proposed to add information of the entire superpixel instead of only using the information of the center point. The evaluation results demonstrate that the proposed superpixel segmentation system achieves boundary recall and under-segmentation error comparable to the original SLIC superpixel segmentation system. For label consistency, the proposed system is more than 0.02 higher than the original system.
Can you read lips with a masked face?

We have been working on lip-reading that estimates the content of utterances using only visual information. When most people started wearing masks due to the coronavirus pandemic, several people asked us if using machine-based lip-reading even when wearing a mask was possible. Taking this as a research question, we worked on word-level lip-reading for the world’s first masked face image. The utterance scene dataset when wearing a mask is not open to the public, so we developed our dataset, face detection for masked face images, facial landmarks detection, and lip-reading based on deep learning. We collected speech scenes of 20 people for 15 Japanese words and obtained a recognition accuracy of 88.3% as a result of the recognition experiment. This paper reports that lip-reading is possible for masked face images.
A Chatbot Assistant for Reducing Risk in Machinery Design

In this paper, a novel chatbot for risk reduction as an aid during machinery design is presented. The general workflow of the chatbot involves the identification of the hazard described by the user using a neural network model followed by an interactive dialog based conversation, in which the risk reduction measures are outlined. A prototype implementation of the chatbot presents the steps to generate and pre-process the training data for Artificial Intelligence (AI) based models. Different neural network models are trained and evaluated for the proposed risk reduction chatbot. A comparative study is presented by employing an in-depth qualitative and quantitative evaluation. The work presented in this paper shows significant promise in ensuring safety awareness, thereby aiding in implementing functional safety in the early stages of machinery design and development.
External Magnetic Interference Classification in Magnetostrictive Position Sensors using Neuro-Symbolic AI with Log-Likelihood Ratios

Magnetostrictive Position Sensors (MPS) are used for precise distance and velocity measurements. They utilize magnetostriction to generate structure-borne sound waves and work on the basis of Time-of-Flight (ToF) calculations. However, external electromagnetic interference (EMI) can impact the accuracy of these sensors by interacting with the magnetic fields of magnetostriction. To address this issue, a novel hybrid approach utilizing both neural and symbolic AI has been developed to classify the intensity of EMI. This system is based on the combination of Log-Likelihood Ratios (LLRs). This study’s findings are particularly significant for industrial environments with numerous sources of external electromagnetic interference, where precise measurement is critical.
Video Anomaly Detection Using Encoder-Decoder Networks with Video Vision Transformer and Channel Attention Blocks

A surveillance camera has been introduced in various locations for public safety. However, security personnel who have to keep observing surveillance camera movies with few abnormal events would be boring. The purpose of this study is to develop a computerized anomaly detection method for the surveillance camera movies. Our database consisted of three public datasets for anomaly detection: UCSD Pedestrian 1, 2, and CUHK Avenue datasets. In the proposed network, channel attention blocks were introduced to TransAnomaly which is one of the anomaly detections to focus important channel information. The areas under the receiver operating characteristic curves (AUCs) with the proposed network were 0.827 for UCSD Pedestrian 1, 0.964 for UCSD Pedestrian 2, and 0.854 for CUHK Avenue, respectively. The AUCs for the proposed network were greater than those for a conventional TransAnomaly without channel attention blocks (0.767, 0.934, and 0.839).
Hardware-Aware Zero-Shot Neural Architecture Search

Designing a convolutional neural network architecture that achieves low-latency and high accuracy on edge devices with constrained computational resources is a difficult challenge. Neural architecture search (NAS) is used to optimize the architecture in a large design space, but at huge computational cost. As a countermeasure, we use here the zero-shot NAS method. A drawback to the previous method was that a discrepancy of correction occurred between the evaluation score of the neural architecture and its accuracy. To address this problem, we refined the neural architecture search space from previous zero-shot NAS. The neural architecture obtained using the proposed method achieves ImageNet top-1 accuracy of 75.3% under conditions of latency equivalent to MobileNetV2 (ImageNet top-1 accuracy is 71.8%) on the Qualcomm SA8155 platform.