Reviewerly/exHarmony huggingface dataset:
Safety Helmet Wearing Detection System Based on a Two-Stage Network Model

In this study, we proposed a safety helmet-wearing detection system to identify whether workers in hazardous environments such as factories or construction sites wear safety helmets. To reduce the classification loss and maintain the availability of streaming helmet detection, we presented a two-stage network model containing YOLOv5-Small and ResNet-18. The network YOLO is used to detect and crop the regions of the images with heads; then, the cropped images will be sent to ResNet to classify. By taking advantage of the individual strengths of these two models, experimental results show that the detection system indeed provides higher precision results compared to the YOLOv5-Small-only system, especially for the cases of wearing hats but not helmets. Furthermore, we designed an additional alert mechanism for real-world applications to reduce the incidence rate of false alerts and increase the flexibility of wide deployment.
SHIFT15M: Fashion-specific dataset for set-to-set matching with several distribution shifts

Set-to-set matching is the problem of matching two different sets of items based on some criteria. Especially when each item in the set is high-dimensional, such as an image, set-to-set matching is treated as one of the applied problems to be solved by utilizing neural networks. Most machine learning-based set-to-set matching generally assumes that the training and test data follow the same distribution. However, such assumptions are often violated in real-world machine learning problems. In this paper, we propose SHIFT15M, a dataset that can be used to properly evaluate set-to-set matching models in situations where the distribution of data changes between training and testing. Some benchmark experiments show that the performance of naive methods drops due to the effects of the distribution shift. In addition, we provide software to handle SHIFT15M dataset in a very simple way: https://github.com/st-tech/zozo-shift15m. The URL for the software will appear after this manuscript is published.
Detail-Preserving Self-Supervised Monocular Depth with Self-Supervised Structural Sharpening

We propose to further close the gap between self-supervised and fully-supervised methods for the single view depth estimation (SVDE) task in terms of the levels of detail and sharpness in the estimated depth maps. Detailed SVDE is challenging as even fully-supervised methods struggle to obtain detail-preserving depth estimates. While recent works have proposed exploiting semantic masks to improve the structural information in the estimated depth maps, our proposed method yields detail-preserving depth estimates from a single forward pass without increasing the computational cost or requiring additional data. We achieve this by exploiting a missing component in SVDE, Self-Supervised Structural Sharpening, referred to as S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> . S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> is a mechanism that encourages a similar level of detail between the RGB input and the depth/disparity output. To this extent, we propose a novel DispNet-S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> network for detail-preserving SVDE. Our network exploits un-blurring and un-noising tasks of clean input images for learning S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> without the need for either additional data (e.g., segmentation masks, matting maps, etc.) or advanced network blocks (attention, transformers, etc.). The recovered structural details in the un-blurring and un-noising operations are transferred to the estimated depth maps via adaptive convolutions to yield structurally sharpened depths that are selectively used for self-supervision. We provide extensive experimental results and ablation studies that show our proposed DispNetS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> network can yield fine details in the depth maps while achieving quantitative metrics comparable to the state-of-the-art for the challenging KITTI dataset.
Exploiting the Complementarity of 2D and 3D Networks to Address Domain-Shift in 3D Semantic Segmentation

3D semantic segmentation is a critical task in many real-world applications, such as autonomous driving, robotics, and mixed reality. However, the task is extremely challenging due to ambiguities coming from the unstructured, sparse, and uncolored nature of the 3D point clouds. A possible solution is to combine the 3D information with others coming from sensors featuring a different modality, such as RGB cameras. Recent multi-modal 3D semantic segmentation networks exploit these modalities relying on two branches that process the 2D and 3D information independently, striving to maintain the strength of each modality. In this work, we first explain why this design choice is effective and then show how it can be improved to make the multi-modal semantic segmentation more robust to domain shift. Our surprisingly simple contribution achieves state-of-the-art performances on four popular multi-modal unsupervised domain adaptation benchmarks, as well as better results in a domain generalization scenario.
Global Motion Understanding in Large-Scale Video Object Segmentation

In this paper, we show that transferring knowledge from other domains of video understanding combined with large-scale learning can improve robustness of Video Object Segmentation (VOS) under complex circumstances. Namely, we focus on integrating scene global motion knowledge to improve large-scale semi-supervised Video Object Segmentation. Prior works on VOS mostly rely on direct comparison of semantic and contextual features to perform dense matching between current and past frames, passing over actual motion structure. On the other hand, Optical Flow Estimation task aims to approximate the scene motion field, exposing global motion patterns which are typically undiscoverable during all pairs similarity search. We present WarpFormer, an architecture for semi-supervised Video Object Segmentation that exploits existing knowledge in motion understanding to conduct smoother propagation and more accurate matching. Our framework employs a generic pretrained Optical Flow Estimation network whose prediction is used to warp both past frames and instance segmentation masks to the current frame domain. Consequently, warped segmentation masks are refined and fused together aiming to inpaint occluded regions and eliminate artifacts caused by flow field imperfects. Additionally, we employ novel large-scale MOSE 2023 dataset to train model on various complex scenarios. Our method demonstrates strong performance on DAVIS 2016/2017 validation (93.0% and 85.9%), DAVIS 2017 test-dev (80.6%) and YouTube-VOS 2019 validation (83.8%) that is competitive with alternative state-of-the-art methods while using much simpler memory mechanism and instance understanding logic.
RIDNet Assisted cGAN Based Channel Estimation for One-Bit ADC mmWave MIMO Systems

The estimation of millimeter-wave (mmWave) massive multiple input multiple output (MIMO) channels becomes compelling when one-bit analog-to-digital converters (ADCs) are utilized. Furthermore, as the number of antenna increases, pilot overhead scales up to provide consistent channel estimation, eventually degrading spectral efficiency. This study presents a channel estimation approach that combines a conditional generative adversarial network (cGAN) with a novel blind denoising network with a sparse feature attention mechanism. Performance analysis and simulations show that using a cGAN fused with a feature attention-based denoising neural network significantly enhances the channel estimation performance while requiring less pilot transmission.
StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation

Anticipation problem has been studied considering different aspects such as predicting humans’ locations, predicting hands and objects trajectories, and forecasting actions and human-object interactions. In this paper, we studied the short-term object interaction anticipation problem from the egocentric point of view, proposing a new end-to-end architecture named StillFast. Our approach simultaneously processes a still image and a video detecting and localizing next-active objects, predicting the verb which describes the future interaction and determining when the interaction will start. Experiments on the large-scale egocentric dataset EGO4D [17] show that our method outperformed state-of-the-art approaches on the considered task. Our method is ranked first in the public leaderboard of the EGO4D short term object interaction anticipation challenge 2022 and it is the official baseline for the 2023 one. Please see the project web page for code and additional details: https://iplab.dmi.unict.it/stillfast/.
A Comparative Sentiment Analysis about HIV and AIDS on Twitter Tweets Using Supervised Machine Learning Approach

Sentiment analysis plays a crucial role in understanding public attitudes towards important social and political issues discussed on social media platforms. This study focuses on analyzing sentiment in Twitter data related to HIV and AIDS, employing a supervised machine learning approach. The primary objective is to compare the effectiveness of various machine learning algorithms in categorizing the sentiment of tweets. The researcher collected and pre-processed a dataset of tweets using sentiment analysis methods from TextBlob and Vader. Three classification algorithms, namely multinomial Naive Bayes, support vector machine, and logistic regression, were evaluated for their performance in sentiment classification. Our findings demonstrate that support vector machine coupled with an n-gram model achieved the highest accuracy, reaching 99% in sentiment classification. This comparative analysis makes a novel contribution to the field of sentiment analysis, specifically in the context of Twitter data related to HIV and AIDS. The insights gained from this study provide valuable implications for understanding public attitudes towards HIV and AIDS on social media and can inform the development of effective public health messaging and policy initiatives pertaining to this critical topic.
Correlation Pyramid Network for 3D Single Object Tracking

3D LiDAR-based single object tracking (SOT) has gained increasing attention as it plays a crucial role in 3D applications such as autonomous driving. The central problem is how to learn a target-aware representation from the sparse and incomplete point clouds. In this paper, we propose a novel Correlation Pyramid Network (CorpNet) with a unified encoder and a motion-factorized decoder. Specifically, the encoder introduces multi-level self attentions and cross attentions in its main branch to enrich the template and search region features and realize their fusion and interaction, respectively. Additionally, considering the sparsity characteristics of the point clouds, we design a lateral correlation pyramid structure for the encoder to keep as many points as possible by integrating hierarchical correlated features. The output features of the search region from the encoder can be directly fed into the decoder for predicting target locations without any extra matcher. Moreover, in the decoder of CorpNet, we design a motion-factorized head to explicitly learn the different movement patterns of the up axis and the x-y plane together. Extensive experiments on two commonly-used datasets show our CorpNet achieves state-of-the-art results while running in real-time.
Just a Glimpse: Rethinking Temporal Information for Video Continual Learning

Class-incremental learning is one of the most important settings for the study of Continual Learning, as it closely resembles real-world application scenarios. With constrained memory sizes, catastrophic forgetting arises as the number of classes/tasks increases. Studying continual learning in the video domain poses even more challenges, as video data contains a large number of frames, which places a higher burden on the replay memory. The current common practice is to sub-sample frames from the video stream and store them in the replay memory. In this paper, we propose SMILE a novel replay mechanism for effective video continual learning based on individual/single frames. Through extensive experimentation, we show that under extreme memory constraints, video diversity plays a more significant role than temporal information. Therefore, our method focuses on learning from a small number of frames that represent a large number of unique videos. On three representative video datasets, Kinetics, UCF101, and ActivityNet, the proposed method achieves state-of-the-art performance, outperforming the previous state-of-the-art by up to 21.49%o.
One-shot skeleton-based action recognition on strength and conditioning exercises

There is a need in the sports and fitness industry for a practical system that can identify and understand human physical activity to enable intelligent workout feedback and virtual coaching. Such a system should be able to classify an athlete’s actions from only limited examples since it is not feasible to collect a large quantity of human data for every action of interest. In this paper, we present SUE-MD, a novel dataset of skeleton motion sequences of seven common strength and conditioning exercises as captured by both a markerless and marker-based motion capture system. We then formulate the one-shot skeleton action recognition problem as a deep metric learning problem. We use the state-of-the-art graph convolutional network (GCN) to project dissimilar actions further away and similar actions closer together in the learned metric space. By training on NTU RGB+D 120, the metric GCN achieves a one-shot performance of 87.4% on all seven never-before-seen actions. In addition, an ablation study reveals the effect of different losses, embedding sizes and augmentations. Our results show that one-shot metric learning method can be used as a means to classify sports actions in a virtual coaching system where users cannot provide many expert examples for the enrolment of new actions.
DeCAtt: Efficient Vision Transformers with Decorrelated Attention Heads

The advent of Vision Transformers (ViT) has led to significant performance gains across various computer vision tasks over the last few years, surpassing the de facto standard CNN architectures. However, most of the prominent variations of Vision Transformers are resource-intensive architectures with huge parameter sizes. They are known to be data-hungry and overfit quickly on comparatively smaller datasets. Consequently, this holds back their widespread usage across low-resource settings, which brings forth the need to develop resource-efficient vision transformers. To this end, we introduce a regularization loss that prioritizes efficient utilization of model parameters by decorrelating the heads of a multi-headed attention block in a vision transformer. This forces the heads to learn distinct features rather than focus on the same ones. Using this loss provides a consistent performance improvement over a wide range of varying scenarios of models and datasets as we show in our experiments, which proves its superior effectiveness.
BEV Approach Based Efficient Object Detection using YoloV4 for LiDAR Point Cloud

Autonomous vehicles and the Advance Drive Assistance System (ADAS) use a number of sensors for the perception of surroundings. One of the frequently used sensors is Light Detection and Ranging (LiDAR). LiDAR gives accurate depth information of the objects around an autonomous vehicle. One of the vital steps of LiDAR-based perception is real-time object detection. Object detection must be performed reasonably well for safety-critical applications with reasonably good accuracy and low latency. Another vital parameter to consider is the detection at a larger distance, the area where LiDAR data becomes sparser. LiDAR-based object detection becomes challenging at more considerable distances as the point cloud becomes sparser. To mitigate this problem, a novel method for point density normalization has been adopted to make the detection less affected by distance. In this paper, an efficient LiDAR point cloud object detection, based on the YoloV4 model, has been proposed, which gives high detection accuracy with the ability to process more than 27 frames per second. For the evaluation of the proposed method of detection, the Kitti dataset was used.
Channel Interference Sensing Transformer for Spread Spectrum Communications with Attention Mechanism

Spread spectrum techniques have been widely adopted in both military and commercial communications because of their anti-interference and anti-interception properties. However, when the interference power is high and the spread spectrum gain is not sufficient to achieve satisfactory system performance, additional anti-interference techniques need to be employed, and the design of the interference sensing and recognition method is a basic prerequisite of effective interference suppression. In this work, considering the superiority of transformer networks in sequence-related tasks, we propose a novel channel interference sensing method with gated transformer and attention mechanism. The proposed network, consisting of embedding, twin-transformer and gating, is designed by introducing the attention mechanism which focuses on the inter-carrier correlation and real-imaginary correlation of the received spectrum. Simulation results demonstrate the superiority of the proposed method versus other baseline competitors.
L1BSR: Exploiting Detector Overlap for Self-Supervised Single-Image Super-Resolution of Sentinel-2 L1B Imagery

Figure 1.L1BSR produces a 5m high-resolution (HR) output with all bands correctly registered from a single 10m low-resolution (LR) Sentinel-2 L1B image with misaligned bands.Note that our method is trained on real data with self-supervision, i.e. without any ground truth HR targets.
Minimizing Energy Consumption for Decentralized Federated Learning Using D2D Communications

Federated learning (FL) is a promising distributed machine learning technique for building inference models over wireless networks due to its ability to maintain user privacy and reduce communication overhead. In this paper, we consider minimizing the energy consumption of a device-to-device (D2D) network while maintaining the convergence rate of FL subject to its time constraint. In the considered D2D network, each device has limited transmission range and is connected partially to other devices in the network. A group of devices can form a cluster and one of these devices is judiciously selected as a local aggregator (LA) to aggregate the local models of other devices in the cluster. Leveraging the nature of D2D communications, we exploit the devices that are located at the conflict zones of LAs. As such, the LAs can disseminate their local aggregated models among them. Towards this goal, a joint optimization problem, considering scheduling the devices to the LAs and computation frequency allocation of the devices, is presented. In order to solve this NP-hard problem, an iterative solution is devised. Particularly, we decompose it into two sub-problems, namely, LAs selection and device scheduling sub-problem and computation frequency allocation sub-problem. By solving theses sub-problems iteratively, a FedD2D (federated learning with D2D communications) scheme is proposed. MATLAB simulations are conducted to verify the effectiveness of the proposed FedD2D scheme over FL conventional schemes.
Dual Attention Poser: Dual Path Body Tracking Based on Attention

Currently, mixed reality head-mounted displays tracking the full body of users is an important human-computer interaction mode through the pose of the head and the hands. Unfortunately, users’ virtual representation and experience is limited due to high reconstruction error when simple transformer network architecture is applied. In this paper, we present a novel model, named Dual Attention Poser, which can learn the whole body reconstruction at a high accuracy. The proposed model consists of three key modules. Among them, dual-path attention encoder is designed to extract feature of the sparse signals. Cross attention mixer module enable the fusion of representation in the double path. Attention-gated-mlp decoder is applied to decode the hidden feature from the sparse input through attention gate. Test results on the AMASS dataset show that Dual Attention Poser can reduce the error by up to 18.2% in comparison with the state-of-the-art results.
Ante-Hoc Generation of Task-Agnostic Interpretation Maps

Existing explainability approaches for convolutional neural networks (CNNs) are mainly applied after training (post-hoc) which is generally unreliable. Ante-hoc explainers trained simultaneously with the CNN are more reliable. However, current ante-hoc explanation methods mainly generate inexplicit concept-based explanations tailored to specific tasks. To address these limitations, we propose a task-agnostic ante-hoc framework that can generate interpretation maps to visually explain any CNN. Our framework simultaneously trains the CNN and a weighting network - an explanation generation module. The generated maps are self-explanatory, eliminating the need for manual identification of concepts. We demonstrate that our method can interpret classification, facial landmark detection, and image captioning tasks. We show that our framework is explicit, faithful, and stable through experiments. To the best of our knowledge, this is the first ante-hoc CNN explanation strategy that produces visual explanations generic to CNNs for different tasks.
A Proposed Quantum Classification Algorithm for Symbol Detection with Noisy Observation

Quantum machine learning (QML) and quantum communication (QC) are emerging areas of research for complexity-efficient data processing, information retrieval, system, and simulations. Towards next-generation vehicular technology, quantum circuits and algorithms are critical in providing faster and more reliable solutions. Here, we show a modified quantum K-nearest neighbour (QKNN) algorithm for symbol detection in the communication system. In the proposed algorithm, we embed a quantum swap-test circuit for the distance computation between unknown symbols with known training symbols with a memory-efficient strategy, which eliminated quantum search operations for the small-search space such as 16-QAM symbols. The proposed quantum circuit is simulated on an IBM quantum machine and the numerical results are shown to demonstrate the efficacy of the proposed framework. The classification accuracy as a performance metric is shown and the improvement of quantum resources in terms of gate complexity is discussed.
Abstract Visual Reasoning Enabled by Language

While artificial intelligence (AI) models have achieved human or even superhuman performance in many well-defined applications, they still struggle to show signs of broad and flexible intelligence. The Abstraction and Reasoning Corpus (ARC), a visual intelligence benchmark introduced by François Chollet, aims to assess how close AI systems are to human-like cognitive abilities. Most current approaches rely on carefully handcrafted domain-specific program searches to brute-force solutions for the tasks present in ARC. In this work, we propose a general learning-based framework for solving ARC. It is centered on transforming tasks from the vision to the language domain. This composition of language and vision allows for pre-trained models to be leveraged at each stage, enabling a shift from handcrafted priors towards the learned priors of the models. While not yet beating state-of-the-art models on ARC, we demonstrate the potential of our approach, for instance, by solving some ARC tasks that have not been solved previously.