Reviewerly/exHarmony huggingface dataset:
AGT: Channel Pruning Using Adaptive Gradient Training for Accelerating Convolutional Neural Networks

Channel pruning is a widely used approach that can efficiently reduce inference time and memory footprint by removing unnecessary channels in convolutional neural networks. In previous studies, channel pruning based on sparsity training was performed by imposing ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> regularization on the scaling factor in batch normalization, and thereafter removing channels/filters below the predefined threshold. However, because channel pruning based on sparsity training imposes ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> penalty on all scaling factors and uses the deformed gradient, an accuracy drop is inevitable. To address this problem, we propose a new sparsity training method referred to as adaptive gradient training (AGT). The proposed AGT can create a compact network without performance degradation using the original gradient to the extent possible without ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> penalty usage. The proposed AGT can reduce the FLOPs of MobileNetV1 by 71.7% on the CIFAR-10 dataset while achieving an accuracy improvement of 0.04%. Consequently, the proposed method outperformed existing channel pruning methods for all datasets and models.
Three-Dimensional Residual Dense Network For High Dynamic Range Imaging

Recently, various deep learning-based high dynamic range (HDR) imaging methods have been proposed to overcome the performance limitation of simply merging or fusion of multiple low dynamic range (LDR) images. However, early deep learning-based methods did not fully exploit the inter-image relationship between different exposures. T o consider inter-image features without the dataset dependency, this paper presents a novel HDR method using three-dimensional residual dense network (3D RDN). We first reconstruct a set of differently exposed LDR images in video format. Next, we take the reconstructed video as an input to the 3D RDN, which generates an HDR image as output. Experimental results show that the proposed method provides higher score compared with state-of-the-arts HDR methods. To the best of authors knowledge, the proposed work first adopted the 3D RDN in the field of HDR imaging using more than three LDR images without any additional pre-or post-processing.
Person Re-identification Method Using Text Description Through CLIP

A typical person re-identification (Re-ID) system works by taking person image as a query to find the most similar person among the images inside the gallery. From this, the system’s performance depends heavily on the quality of the query image. We see the hint to overcome this limitation in recent surprising progress in multi-modal learning between vision and language. In this context, this paper proposes a person re-identification method that utilizes text guidance via the Contrastive Language-Image Pre-training (CLIP). To fully utilize CLIP, we show how to transfer their knowledge to person Re-ID network. Experimental results prove the superior performance of our method on person Re-ID.
A Survey of Visual Commonsense Generation Task

The task of extracting commonsense from data has received a lot of attention, and various research on commonsense tasks is conducted. This paper provides an overview of the visual commonsense generation(VCG) tasks using visual and language data. First, we will describe the differences between VCG tasks and other tasks that use commonsense such as reasoning, question, answering, and etc. Next, we will look at what datasets are available and what models are being used in relation to the VCG task. Finally, we will explain the limitations that are in the VCG task studied so far and will discuss future research directions in VCG task. This survey will provide an introduction to VCG task and a guide to practitioners.
Image Stitching Method for Surround View Image without Seamline

The surround view system delivers the surrounding environment to the driver in a top-down manner so that the driver can recognize blind spots. In this system, image stitching is combining camera images facing different directions into a single image. As an easy-to-access method, there is a method of warping through homography by extracting and matching the features of the images to be stitched. However, this method has a limitation in that stitching is not performed properly if the geometric distortion is severe. In order to solve this problem, a method of estimating a seam and stitching images is being studied in many existing methods, and a representative method is a method using a distance matrix using the difference of gradient components. However, this method has a problem of falling into the local minima and causing a loop phenomenon. To solve this problem, we propose a seamline estimation method using ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> -norm based gradient priors. The optimal seam can be found by obtaining the energy matrix through ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> -gradient priors and estimating the distance matrix through this. In this method, improved seam estimation is possible because the shortest distance is calculated as the point where the rate of change between the two images is minimized.
A Study on the Application of OpenPose for the Prevention of Collision of VR HMD Wearers

In this paper, a pose estimation was implemented using the input image of the HMD front camera to prevent collision accidents of VR(Virtual Reality) HMD(Head Mounted Display) wearer. In the experimental environment, the OpenPose library was applied to the Vive Pro Eye front camera using the Unity engine. Through this, we propose a method for human-to-human collision of HMD users whose vision is blocked.
Image Enhancement for High-Resolution Visual Contents

This paper proposes an image enhancement method using gamma neural networks and exponential transformation. When acquiring an image, degradation occurs in very many imaging systems, and the quality of the image acquired by surrounding environmental factors decreases due to the combination of deteriorating elements. Alternatively, work that facilitates post-treatment may be performed by artificially deteriorating for post-treatment directly. However, if the information on these additional tasks is not known, there is a problem that the post-processing process is expensive or additional degradation occurs. To solve this problem, this paper uses a neural network that estimates gamma maps through residual learning for images that require post-processing, and finally applies exponential transformations to perform contrast improvement. The contrast improvement method proposed through the experimental results provides an image with less color distortion compared to the existing method.
Empirical Analysis of Side-Channel Attack Resistance of HLS-designed AES Circuits

Many IoT devices have been compromised by side-channel attacks. Among them, power analysis attack, which identifies the secret key of a circuit by analyzing the power consumption of the target device, is well-known. Although there is generally a trade-off between performance and hardware cost (resources) in circuits, the correlation between performance and security is not known. This study investigates the relationship between side-channel attack resistance, performance (clock cycles) and hardware resources of AES circuits. Six AES circuits with different performance are designed by high-level synthesis. T-test is employed to assess side-channel attack resistance. The results show that there is a correlation between side-channel attack resistance, performance and hardware resource usage, but the relationship varies depending on the definition of side-channel attack resistance.
Machine Learning Approaches to Facial Recognition: A Survey

Due to the rapid development of technology, face recognition has received much attention recently. The face continues to be the most challenging study topic for experts in computer vision and image processing since it has unique characteristics that must be recognized as an entity. This survey explores the most challenging face aspects, including position change, age, lighting, and occlusion. They are considered essential elements in facial recognition systems when used with facial images. The current state of face detection methods and techniques is also examined in this paper. These methods and techniques include Eigenface, Artificial Neural Network (ANN), Support Vector Machine (SVM), Principal Component Analysis (PCA), Independent Component Analysis (ICA), Gabor Wavelets, 3D morphable models, and Hidden Markov Model (HMM). However, the goal of this study is to give a comprehensive assessment of the literature on face recognition and its applications. After a thorough discussion, the most important findings are presented.
Eight-bit Quantization for Super-Resolution Networks and Its Hardware Implementation

This work presents an 8-bit quantization scheme for super-resolution (SR) networks and its hardware implementation. Quantizing SR models to run in 8-bit is a non-trial task, frequently leading to non-negligible restoration performance degradation. To address the problem, we first present a quantization scheme that mitigates accuracy drop by effectively capturing the distributions of both weights and activations. Second, the proposed scheme is implemented using Verilog HDL and the design is synthesized on Synopsys Design Compiler using TSMC 65 nm library. Lastly, experimental results illustrate the effectiveness and efficiency of the proposed method.
AI Feedback Architecture of Video Surveillance System

The learning capacity of general deep learning models for object detection would not be large enough to represent real-world scene dynamics, and thus such models would be weak to ‘unseen’ data due to environmental changes. Therefore, in this study, we propose a new method to continuously improve the object detection algorithms by applying negative and positive learning mechanisms, especially for intrusion detector in video surveillance systems. By applying an iterative process where the current model is updating using new incoming data with a state-of-the-art model in a continual process of adaptation. The experimental results in various challenging videos for real video surveillance systems demonstrate that the proposed method offers a significantly improved algorithm accuracy with a low complexity, thus it is adapted for real-world systems.
Please Be Nice: A Deep Learning Based Approach to Content Moderation of Internet Memes

The internet has grown throughout time from a means for information distribution to a multipurpose tool that allows people to communicate, meet, and interact with one another. In socializing online, sending images with cultural and relatable context, otherwise known as memes, has become a common practice. Memes are sent everywhere, from forums and posts to even private messages. They are famous for converging thoughts, current events, or symbols creatively and colloquially. It combines text with a visual graphic and conveys many emotions, including hatred, which can harm other netizens. As a tool for communication, it can be used to convey humor or even as a tool for cyberbullying through hateful speech. Since memes are sent in image form, filtering out hateful speech in the form of memes can be difficult. To address this rising issue, researchers explore the use of deep learning approaches to detect hate speech on multimodal memes as a systematic approach for eliminating hate speech. The model performed well in this investigation, with an overall accuracy of 62.5%. When identifying pictures, it also has a high proportion of precision (71.37%). This brings the suggested model’s ability to detect hate speech in memes to a close.
A Unified framework based on Large-scale Momentum Contrastive learning for Text-Video Retrieval

Large-scale vision-language representation learning has shown dramatic advances in image-text retrieval studies. On top of that, it is clear that the CLIP (Contrastive Language-Image Pre-training) [1] has contributed to this field in that it succeeded pre-training on tremendous web multimedia data with leveraging contrastive loss, one of the most competitive loss for multimodal learning. CLIP4CLIP [2] utilized the pre-trained CLIP weight to appeal its performance of learning video representations with textual supervision with additional post-pretraining on Howto100M [3]. However, there is still room for elevation in CLIP4Clip from perspective of contrastive learning - large enough instances of negative samples help to learn better representations, as the batch size in each training batch getting larger, the more chance of the model to be allowed to contrast more semantic representations. In the same queue of this point of view, we propose CwMTVR combining Momentum Contrast (MoCo) [4], cross-modal momentum contrastive learning framework into CLIP4Clip. Experimental results prove that the CwMTVR model achieve competitive results in text-video retrieval on MSVD dataset.
IDS-Extract:Downsizing Deep Learning Model For Question and Answering

In recent years, Question-answering systems are extensively used in human-computer systems, and the accuracy rate on a large scale is increasing. However, in actual deployment, a large number of parameters are often accompanied by a large amount of memory and long-term processing requirements. Therefore, compressing the data of the model, reducing training time, memory, becomes more and more urgent. we aim to resolve issues: IDS-Extract dynamically sized data to support models and devices with different memory. The proposed technique does efficient data extraction, segments that are not meaningful for model learning on the original dataset and output multiple datasets of adaptive size followed by target training based on model size. We leverage techniques in IG(Integration Gradient), DPR, and SBERT to improve localization performance for answer positions. We compare the model performance of SQuAD and the data set reduced by the IDS extraction technique, and the results prove that our technique can train the model more targeted and obtain higher performance evaluation. We prove that this method has successfully passed the sanity check, and can be directly applied to emotion recognition, two-classification, and multi-classification fields.
Diagnose Label Errors for 3D Object Classification

3D object classification is widely used in many real-life scenarios and has recently become a popular research area. Meanwhile, 3D datasets tend to be larger and more complex, increasing the difficulty of labeling. Since label errors in test data affect the accuracy of model evaluation and analysis, this study proposes a multi-view post-training label refinery protocol to clean the label errors in test data. Experimental results on ModelNet40 show that 4.58% of test samples are selected as label-error candidates and 29.2% of them are relabeled. On this basis, this study analyzes the weakness of ModelNet40 and provides suggestions for future dataset construction.
Machine Learning-based Signal-to-Noise Ratio Estimation using Amplitude Frequency Vector

In this paper, we propose a new SNR estimator using a machine learning model (MLSE) that has trained the amplitude pattern of data symbols. In order for the neural network to estimate the SNR, the received data symbols are converted into a kind of histogram, an amplitude frequency vector (AFV), depending on the amplitude value. The machine learning model is trained to match the pattern of the AFV to the SNR, and as a result, the MLSE can estimate the SNR with a very high accuracy of mean squared error (MSE) below 0.01 even in very low SNR region. Unlike conventional SNR estimation techniques that utilize additional information including pilot signals, the proposed SNR estimator uses only data symbols, so there is no signaling overhead. In addition, since it uses a machine learning model, its computational complexity is very low.
InvHDR: Inverse Tone Mapping With Invertible Neural Network

In general, many cameras capture a relatively small range of luminosity due to the price of high-performance camera sensors. We can compensate this by creating high dynamic range (HDR) images using multi-stack fusion, which combines information from multiple low dynamic range (LDR) images of various dynamic ranges. However, ghost artifact is inevitably caused by moving objects when acquiring multiple images. Therefore, many state-of-the-art methods create pseudo-image stack of multiple dynamic ranges from one single LDR image and fuse them to a HDR image. In this paper, we use invertible neural network to improve the existing methods and propose a conditional network structure with improved performance on multiple indicators.
Whole-body Human Mesh Reconstruction with Transformer

Whole-body human mesh reconstruction aims at localizing mesh vertices of all human parts, including body, hands, and face simultaneously. The challenges of this problem are small image sizes of hands and face and smoothly connecting reconstructed 3D body, hands, and face meshes. To resolve these challenges, we present a transformer-based architecture, WholeTransNet, a novel whole-body mesh reconstruction network. Our WholeTransNet outperforms previous whole-body mesh reconstruction methods.
GPU Acceleration of Chinese Remainder Theorem for Fully Homomorphic Encryption

Fully Homomorphic encryption (FHE) is an encryption technique capable of performing data operations without decryption operations on encrypted data. With the development of the Internet and AI technology, concerns about personal information have increased. Therefore, the characteristic of being able to operate in the encrypted state of homomorphic Encryption is suitable for application to personal information security technologies. FHE enables data processing while maintaining security between third parties. However, because the calculation time of FHE is very slow, the high computational cost of homomorphic encryption must be addressed before it can be applied to commerce. We focused on multiplication, the slowest, and the main operation of the homomorphic encryption scheme, Cheon, Kim, Kim, and Song (CKKS). In this paper, we accelerate multiplication operations by assigning blocks and threads of GPUs to FHE polynomials. By implementing Chinese remainder theorem (CRT) operations, one of the detailed kernels of multiplication on the GPU, We achieved about 4x the speed improvement over the CPU.
DWT+DWT: Deep Learning Domain Generalization Techniques Using Discrete Wavelet Transform with Deep Whitening Transform

Recently, there is a growing demand for a deep learning framework with robust generalization performance in real-world domains, such as an autonomous driving environment. The existing domain generalization methodologies for convolutional neural networks have been designed to actively utilize the feature map with the generative model or normalization techniques to distinguish domain-specific information. However, augmented images are essential for measuring style sensitivity. This study shows that style information can be extracted from an original image through color space separation and frequency decomposition without a separate augmented image. Therefore, it can be used as a method independent of existing network models. The proposed method shows an mIoU improvement by 1.54% compared to the existing method in the semantic segmentation model trained using urban scene datasets.
Multi-view stereo with recurrent neural networks for spatio-temporal consistent depth maps

Depth estimation methods based on deep learning have been studied to improve depth estimation accuracy. However, obtaining inter-frame consistency in depth maps in video depth estimation remains a challenge. Therefore, we proposed an application methodology for spatio-temporal consistency enhancement in video depth estimation based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In other words, the convolutional long-short term memory (ConvLSTM) module was added to the decoder of depth estimation network to enable the use of the information from the previous frames. Additionally, the one-stage learning process was implemented to ensure ease of training. In conclusion, we experimentally show that the proposed method can achieve not only improved accuracy also consistency between depth map frames.
Audio-to-Facial Landmarks Generator for Talking Face Video Synthesis

Audio driven talking face methods have been studied to process the accuracy lip synchronization. However, how to create movement of head poses and personalized facial features is a challenging problem. In order to solve this problem, it is necessary to identify the context based on the audio, create the head pose and lip motion, and synthesize the personalized face. We introduce a facial landmark generation method including audio-based head pose and lip motion using an audio transformer. The audio transformer extracts audio features containing contextual information and creates generalized head pose and lip motion landmarks. In order to synthesize personalized features on the generated landmarks, a talking face video is generated by applying the method learned through meta-learning. With just a few single images, even unknown faces can be spoken in the audio you want. In addition, the proposed method is applicable to various languages, and enables photo-realistic synthesis and fast inference.
Data Collection Framework Using a Lightweight Blockchain

Blockchain protocol has emerged in modern technologies in recent years. Significantly, the data collection frameworks are one of the most demanding fields of blockchain adaptation. However, the traditional approach relies on heavy computations, which require high-computing servers that cannot be affordable by all providers. This paper proposes a reliable and secure data collection technique, BSave, that employs a lightweight blockchain suitable for low-computing devices. BSave stores the data in a key-value database and enables two-way encrypted security on both keys and values. Compared to traditional approaches, the BSave proved its effectiveness in data-intensive environments.
One-Shot Face Reenactment with 2D Facial Landmark Conditional Normalizing Flow

Normalizing Flow (NF) has gained growing popularity in various image generation tasks. In this work, we develop a new method that enables the NF to control face generation, which has not been studied yet. To do so, we introduce several loss functions to facilitate stable training and inference while controlling face generation given a facial landmark. In our experiments, we evaluate the performance of the proposed method and show the capability of the NF in controlling the face generation task.