Reviewerly/exHarmony huggingface dataset:
Achieving a Lawfully-Secure Audio Recording Framework using Consumer Electronics

The wide adoption of Consumer Electronics have boosted several fields including entertainment, communications and recreation. The need of communication took advantage by the spread of a plethora of consumer electronics devices, such as modern smartphones. Such devices are often used for sharing multimedia contents among group of people. This paper illustrates a practical application to be used for sharing sensible multimedia information (such as audio or video data) among members of a organization, such as law enforcement, judicial members of a court, lawyers, and so on. The main focus is to keep secure the exchanged data among members of a given organization, available all the times and hidden from unauthorized personnel. The second objective is to prevent any abusive access to the data from some authorized members of the organization without others consent or knowledge. In this way, a protected multimedia file cannot be decrypted only by a one person since the decryption key can be reconstructed if and only if a subset of the entire set of participants to the group agrees
Dynamic Feature Fusion for Visual Object Detection and Segmentation

Feature fusion is a key process of integrating multiple features in deep neural networks (DNN). The mainstream method in the literature is based on the Feature Pyramid Network (FPN), where the learned parameters about feature fusion is fixed after the training process. That is, how the multiple features will be fused is independent from the embedded characteristics of the input data, making the feature fusion process less flexible especially for the object categories less seen in training data. Therefore, this paper proposes a novel feature fusion mechanism, called dynamic feature fusion. With this mechanism, a model can automatically learn and select the appropriate way of feature fusion to provide prediction heads with more effective and flexible input features depending on the characteristics of input data.
iPhone 240fps Video Dataset for Various Model Training Tasks

High frame rate videos are recently used for training a variety of image processing tasks such as image/video deblurring and frame interpolation. Yet, the number of publicly available datasets with high frame rate videos are still limited. In this paper, we propose the iPhone 240fps dataset, a dataset which consists of real world 240fps videos. We conduct experiments with deblurring and frame interpolation models to show the effectiveness of this dataset. We also use this dataset to conduct experiments with a state-of-the-art video super-resolution model. We take advantage of the flexibility to change the frame rate of the dataset and test which fps is best suited for training the model. We believe this dataset can make contributions to future works in image and video processing. The dataset is publicly available on our cloud storage <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">†</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">†</sup> https://waseda.box.com/s/4ihxczdiythmeml38ndgsq4l1q90vnpp:
Anchor-Based Detection for Natural Language Localization in Ego-Centric Videos

The Natural Language Localization (NLL) task aims to localize a sentence in a video with starting and ending timestamps. It requires a comprehensive understanding of both language and videos. We have seen a lot of work conducted for third-person view videos, while the task on ego-centric videos is still under-explored, which is critical for the understanding of increasing ego-centric videos and further facilitating embodied AI tasks. Directly adapting existing methods of NLL to ego-centric video datasets is challenging due to two reasons. Firstly, there is a temporal duration gap between different datasets. Secondly, queries in ego-centric videos usually require a better understanding of more complex and long-term temporal orders. For the above reason, we propose an anchor-based detection model for NLL in ego-centric videos.
Multi-Layer Attention-based State Representation for the Reinforcement Learning of Visual Servoing

In this paper, we propose multi-layer attention-based state representation for robotic visual servoing using reinforcement learning. It is challenging for a robot to manipulate an object that it has never been trained on, which can occur in real use cases. Multiple attention layers allow the model to extract the spatial features of the object in the scene from high to low resolution in latent space. Using the extracted features allows the robotic controller model to learn more robust trajectory planning. We evaluated and compared the robustness of our method with that of other methods on a fetch-reach task in a simulation by randomly scaling the size of the target object. The results demonstrated that our method yielded the highest success rate, with the most efficient trajectory planning, which indicates that our state representation model provided more useful information that enabled visual servoing to perform more robustly for unseen objects.
Intelligent Candidate Space Identification and Optimization for Video Advertisement Implant

The rapid development of video-supporting consumer electronic devices has significantly promoted the ubiquitous video play. This has encouraged advertisers to seamlessly embed advertisements (ads) in video scenes as one of their marketing strategies. In this paper, we propose our SalAd framework for identifying candidate spaces for video ads implant. By taking the advantage of semantic segmentation, saliency detection, and corresponding optimization, SalAd can intelligently identify multiple candidate spaces while achieving semantic consistency, attractiveness, and reinforced commercial values. To train the SalAd model, we have also constructed an initial dataset containing 1020 scenes with manually labeled candidate spaces. Our experimental results show that SalAd can be compatible with various segmentation neural networks and can intelligently identify candidate spaces.
Head Motion Detection and Compensation for Visible-Spectrum Gaze Tracking Systems

The head motion detection and compensation mechanisms significantly improve the final system precision/accuracy of a gaze tracker. But the problem becomes more difficult for a visible-spectrum gaze track compared with the one based on an infra-ray (IR) lighting system for the lack of the IR light sources as the reference locations. This paper presents a robust detection as well as a compensation mechanism for a visible-spectrum gaze tracker by using more stable feature points on the entire head image. The limbus circle matching module dynamically adjusts the eyeball model and calculates the moving distance of the head. The experimental results show that the compensation mechanism further improves the system performance of the gaze tracker when the user moves their head.
Adaptive Margin-based Contrastive Network for Generalized Zero-Shot Learning

Generalized zero-shot learning is a challenging problem that aims to recognize images from seen and unseen classes. Recent methods are costly and time-consuming or have a bias problem. To tackle this problem, we proposed an adaptive margin-based contrastive network that aims to distinguish similar classes in generalized zero-shot learning. The proposed method employs the architecture of transferable contrastive network to classify unseen classes and adaptive margin to transfer discriminative knowledge. Experiments on the AwA2 dataset demonstrate competitive results against state-of-the-art benchmarks.
FPGA Realization of Lane Detection Unit using Sliding-based Parallel-Segment Detection for Buffer Memory Reduction

With the development of various chips, such as VLSI chips, and the development of semiconductor technology and artificial intelligence, autonomous driving technology is advancing daily. Lane recognition technology, which can be considered the most important in the implementation of autonomous vehicles, requires a large amount of computation and processing time because data must be received from a camera attached to the vehicle and processed in real time. To design a more efficient and implementable lane recognition algorithm, we proposed a method to reduce the usage of buffer memory by using parallel operation. Most of the boards used in autonomous vehicles are lightweight, so the lane recognition algorithm is also lightweight to use a minimum library. First, after reading the image, canny edge-detection is executed through grayscale conversion, Gaussian smoothing, a Sobel operator, non-maximum suppression, and hysteresis in parallel. The lane is inspected using the Hough transform as an input to the image with the edge detected by canny edge-detection. Due to parallel operation's nature, the effect is insignificant when a single image input is received, but the operation is more efficient when multiple images are input in real time. We used a relatively low-level C language for efficiency and processed images with loops and operations.
Implementation of Anti-quantum Communication System using Software-Defined Radio

In recent years, the rapid development of quantum computer has posed great challenges in the security of traditional wireless communication. This paper proposes a future-oriented high security anti-quantum wireless communication system, which can be applied to the mobile devices of Internet of things with constrained resources, and give them longer service life and flexibility on the basis of ensuring their security. Particularly, the Rainbow anti-quantum signature algorithm is implemented in Software-Defined Radio (Zedboard and AD9361 RF modules) platform and accelerated by Field Programmable Gate Array (FPGA) hardware of Zedboard. The experimental results show that the time of signature generation and verification is shortened significantly under the limitation of the consumption of computing resources. What's more, the proposed anti-quantum wireless communication system has better reconfigurability through flexible RF front-end configurations.
A Novel Face Recognition Using Specific Values from Deep Neural Network-based Landmarks

The detection of facial landmarks has been an ongoing research topic for the past decade as it is used in facial recognition, facial expression analysis, and security purposes. This paper proposes a new face recognition algorithm that uses deep learning-based facial landmark detection algorithms to extract key features from images. Using the landmarks obtained from the applied deep neural network, three different features are extracted using specific values and used for face recognition. The use of specific values (SV) for facial recognition is the novelty of this work. Three specific values namely the cosine distance, angles and areas are derived from the coordinates of the landmarks. The recognition rates of faces using the extracted landmarks and SVs as the proposed features are evaluated through several experiments. Of the proposed features, the area provides the best result. Also, to investigate the effect of increasing the number of landmark points on the proposed face recognition rate, the MediaPipe face mesh algorithm is utilized. With the same chosen SVs, the recognition rate results are discussed when recognition was carried out with the increased number of landmarks.
Learning Face Forgery Detection in Unseen Domain with Generalization Deepfake Detector

Face forgery generation algorithms have advanced rapidly, resulting in a diverse range of manipulated videos and images which are difficult to identify. As a result, face manipulation using deepfake technique has a significantly increased societal anxiety and posed serious security problems. Recently, a variety of deep fake detection techniques have been presented. Convolutional neural networks (CNN) architecture are used for most of the deepfake detection models as binary classification problems. These methods usually achieve very good accuracy for specific dataset. However, when evaluated across datasets, the performance of these approaches drastically declines. In this paper, we propose a face forgery detection method to increase the generalization of the model, named Generalization Deepfake Detector (GDD). The Generalization Deepfake Detector model has ability to instantly solve new unseen domains without the requirement for model updates.
Two-Step Self-Supervised Pretraining Approach for Machinist Hand Gesture Recognition

We propose a two-step self-supervised pretraining approach for hand gesture recognition on an assembly line. Obtaining adequate company-specific data from an assembly line for machine learning is difficult because of the frequent need for confidentiality and the cost of annotation. To overcome this quantitative limitation, we propose an approach that performs self-supervised pretraining in two steps using data that are similar to the target data and relatively easier to obtain. In the experiment, the machinist dataset, which is a subset of the Yale Human Grasping dataset, was used as hand gesture data from an assembly line. By comparing different pretraining method and dataset, two-step self-supervised pretraining yielded highest accuracy, showing the improvement obtained by our method. The result indicates that our method has the potential to improve specific hand gesture recognition without expensive data preparation.
Action Recognition Using Multi-stream 2D CNN with Deep Learning-Based Temporal Modality

Video action recognition requires accurate analysis of motion information along with spatial information of an object. In other words, it is necessary to learn both temporal and spatial information. In many deep learning-based action recognition methods, temporal and spatial information are extracted by a multi-stream network, where the temporal stream network analyzes the motion information using mathematical operations. In this paper, we present an action recognition method using a multi-stream network with a deep learning-based temporal relation module, which extracts motion information for the entire video in the temporal network path. The proposed method significantly increases the accuracy of action recognition using attached modules in front of the 2D CNN and late fusion with another network path. Owing to the proposed temporal stream network without additional mathematical operations, we could greatly reduces the amount of computation. As a result, the proposed method is suitable for a wide range of real-time visual action recognition tasks.
A Lightweight Multimodal Learning Model to Recognize User Sentiment in Mobile Devices

Communications through video/audio calling and text messaging has seen a meteoric rise over the last few years. As we progress towards the advanced wireless technology (i.e., 5G), the quality of call and network low latency communication provides better quality of services (QoS). This allows for addition of new features that would provide users with a more personalized experience such as providing sentiments analysis of users through mobile (sender/receiver) devices over text messaging or video calls or both. However to perform sentiment analysis requires Deep Neural Networks (DNN) model to extract multimodal data (such as text, audio and video) features which is very heavy to deploy in mobile devices due to its limited computational capabilities. In the past, various research works has been performed to develop multimodal feature extraction mechanisms, however none of the multimodal models are suited to be deployed in mobile devices for practical applications. To mitigate these issues, we propose a light weight multimodal learning model called Tri-Feature Fusion which can be easily deployable in mobile devices for client-server communications. The Tri-Feature Fusion model extracts the feature vectors of each modality and generate a single multimodal feature vector from the data generated from the call and perform sentiment analysis on each particular mode. We also have developed a light weight neural network for Tri-Feature Fusion to perform the sentiment analysis using the extracted features for real-time audio, video and text data displayed at clients and receiver mobile devices. We conduct extensive experiments to showcase the performance of Tri-Feature Fusion as light weight model in-terms of average time taken for prediction, time taken per step, CPU utilization and size of the models compared with the existing state of art.
Inverted Residual Fourier Transformation for Lightweight Single Image Deblurring

In recent years, encoder-decoder structures are widely used for single image deblurring and successfully restore high quality images. How-ever, FLOPs and the number of parameters tend to increase to restore a high-quality image. Thus, we propose a new lightweight network (IRFTNet) based on DeepRFT. This network has two features to improve performance and lightweight. First, a new backbone called Inverted Residual Fourier Transformation block (IRFTblock) based on inverted residual block is introduced to decrease computational complexity. Second, a new module called Lower Feature Synthesis (LFS) was introduced to efficiently transfer encoder information from lower layers to upper layers. These improvements resulted in a 32.98dB in PSNR on the GoPro dataset, despite approximately half FLOPs and the number of parameters of DeepRFT.
Adaptive Constant Hue Algorithm Design of Color Filter Array

In this paper, we proposed an “adaptive constant hue algorithm design of color filter array.” The method is an extended application of the edge algorithm in the spatial domain algorithm. The main method is to do finer angle calculation on the basis of edge judgment. When the angle judgment is finer, it will be closer to the original image plane. The algorithm we propose is to use multiple angles to perform high-order interpolation adjustments to obtain better image quality. We use standard test images to simulate, and use subjective judgment, PSNR value and computational complexity to compare the effect of the algorithm. The experimental results prove that our proposed method can improve the reconstructed image quality and the color accuracy with a lower computational complexity.
Semi-Supervised Learning Based De-Raining Method for UAV

In recent years, unmanned aerial vehicles (UAVs) have made remarkable progress and are highly expected to play an active role in a variety of fields. However, there are still many concerns regarding the autonomous flight and safety of UAVs. Among them, video image degradation due to rainy weather is a significant problem, regardless the UAV is flying autonomously or remotely controlled. In this work, we propose an efficient learning-based de-raining method using video images of UAVs in rainy conditions. The proposed method creates a de-raining model appropriate for the situation by adding UAV rain images to Syn2Real training data. As a result, the proposed method performs better than the existing de-raining method in PSNR and SSIM.
D2D: Divide to Detect, A Scale-Aware Framework for On-Road Object Detection Using IR Camera

In this paper, to solve the problem of inconsistencies between the predictions in today's SOTA object detection networks, which incorporate the pyramid architecture with multi-level prediction, we proposed a scale-aware framework for IR image-based on-road object detection. The proposed framework uses scale-based attention mechanism to assign responsibilities to each feature levels. With this design, each feature level will focus on detecting a certain range of object scales, thereby minimizing the conflict among the predictions in the final result. Compared to Scaled-YOLOv4 baseline, our proposed method can achieve better performance without increasing FPS on FLIR dataset. The experimental results on RGB image-based object detection datasets also show that our proposed method gives good improvements when applied to RGB images.
Evaluation System for Martial Arts Demonstration from Smartphone Sensor Data Using Deep Neural Networks on Noisy Labels

Smartphones and smartwatches are widespread. Devices have with sensors. Research activities to obtain data from sensors to recognize sports motion and find modifications are on the rise. These studies make it possible to provide movement feedback without a coach. However, a few studies have focused on the correctness of the movement, or which parts of the movement are correct. Herein, we propose a method to evaluate the movements of a martial arts demonstration based on sensor data obtained from a smartphone. In the demonstration, several movements, such as punching and kicking, are performed. The coach considers each movement and evaluates the overall movement comprehensively. Regarding the coach evaluation, we evaluate sensor data by dividing the data and using machine learning. This method was applied to the acceleration data of taekwondo demonstrations. It was shown to reproduce the instructor's score with high accuracy.
Comparison of Wearable Sensor-based Fall Event Detection by 1-D and 2-D Convolutional Neural Networks

Monitoring of the elderly people for their safety and well-being is crucial to help them to age independently in their own homes. Ubiquitous wearable sensors such as accelerometers and gyroscope have been used for monitoring activities of daily life (ADL) and fall. Traditional machine learning algorithms are currently replaced by the emerging deep learning methods to detect fall events and classify ADL. In this paper, fall event detection abilities of two recently proposed CNN architectures for fall event detection are compared. These two architectures have different receptive fields and therefore, discover different number of features for fall event detection. Performance of 1-D and 2-D versions of each of these architectures are compared using the data from two 3-axis accelerometers (large and small dynamic range), and a 3-axis gyroscopes available in a public dataset. 1-D CNN uses raw time signal while 2-D CNN uses the spectrograms of the raw signal as input. Effect of noise on the fall event detection using small dynamic range accelerometer is also studied. It is observed that 2-D CNN provides better fall event detection than 1-D CNN for all the three sensors and fall event detection is slightly better with accelerometers than with gyroscopes.
End to End Optimized Tiny Learning for Repositionable Walls in Maze Topologies

In the context of Tiny Machine Learning, the adoption of Deep Reinforcement Learning has been severely prevented due to the computationally expensive demand of such method. This work proposes the application of a Deep Reinforcement Learning algorithm used to train a tiny Convolutional Neural Network such that can be deployed on a cheap, off-the-shelf Microcontroller Unit (MCU) aimed to solve a physical, electrically actuated tilting maze with repositionable walls. The method proposed to train the agent take advantage the lost games to avoid overfitting on certain trajectories. The adopted neural network has been quantized to 8-bits to fit into the MCU embedded memory, comparing the performances against the non-quantized networks and the variable number of repositionable walls. The trained policy networks achieved encouraging results, with win rates between 87% and 99% depending on the difficulty of the task and the model size of the network, with low inference times of about 10 ms. This result, in principle, enables an End-to-End Learning approach able to autonomously sense from the external world, and then compute and execute the actions to solve the physical maze prototype with variable numbers of repositionable walls across games.