Reviewerly/exHarmony huggingface dataset:
Analyzing Interactions in Paired Egocentric Videos

As wearable devices become more popular, ego-centric information recorded with these devices can be used to better understand the behaviors of the wearer and other people the wearer is interacting with. Data such as the voice, head movement, galvanic skin responses (GSR) to measure arousal levels, etc., obtained from such devices can provide a window into the underlying affect of both the wearer and his/her conversant. In this study, we examine the characteristics of two types of dyadic conversations. In one case, the interlocutors discuss a topic on which they agree, while the other situation involves interlocutors discussing a topic on which they disagree, even if they are friends. The range of topics is mostly politically motivated. The egocentric information is collected using a pair of wearable smart glasses for video data and a smart wristband for physiological data, including GSR. Using this data, various features are extracted including the facial expressions of the conversant and the 3D motion from the wearer's camera within the environment - this motion is termed as egomotion. The goal of this work is to investigate whether the nature of a discussion could be better determined either by evaluating the behavior of an individual in the conversation or by evaluating the pairing/coupling of the behaviors of the two people in the conversation. The pairing is accomplished using a modified formulation of the dynamic time warping (DTW) algorithm. A random forest classifier is implemented to evaluate the nature of the interaction (agreement versus disagreement) using individualistic and paired features separately. The study found that in the presence of the limited data used in this work, individual behaviors were slightly more indicative of the type of discussion (85.43% accuracy) than the paired behaviors (83.33% accuracy).
Circuits for Josephson Parametric Amplification in Quantum Radar

Joesphson parametric amplifiers (JPA), operating in the microwave regime, are a key component for quantum radar. A Josephson junction can combine the functions of local oscillator, mixer, amplifier and detector. For quantum radar based on two-mode squeezed vacuum states (TMSV), the Josephson devices perform squeezed state generation and mixing, for subsequent photon counting and hypothesis testing. In this contribution, we investigate quantum circuit for modeling of JPAs.
Dynamic face imaging: a novel analysis framework for 4D social face perception and expression

Measuring facial expressions is a notoriously difficult and time-consuming process, often involving manual labeling of low-dimensional descriptors such as Action Units (AUs). Computer vision algorithms provide automated alternatives for measuring and annotating face shape and expression from 2D images, but often ignore the complexities of dynamic 3D facial expressions. Moreover, open-source implementations are often difficult to use, preventing widespread adoption by the wider scientific community beyond computer vision. To address these issues, we develop dynamic face imaging, a novel analysis framework to study social face perception and expression. We use state-of-the-art 3D face reconstruction models to quantify face movement as temporal shape deviations in a common 3D mesh topology, which disentangles global (head) movement and local (facial) movement. Using a set of validation analyses, we test different reconstruction algorithms and quantify how well they reconstruct facial “action units” and track key facial landmarks in 3D, demonstrating promising performance and highlight areas for improvement. We provide an open-source software package that implements functionality for easy reconstruction, preprocessing, and analysis of these dynamic facial expression data.
SignNet: Single Channel Sign Generation using Metric Embedded Learning

A true interpreting agent not only understands sign language and translates to text, but also understands text and translates to signs. Much of the AI work in sign language translation to date has focused mainly on translating from signs to text. Towards the latter goal, we propose a text-to-sign translation model, SignNet, which exploits the notion of similarity (and dissimilarity) of visual signs in translating. This module presented is only one part of a dual-learning two task process involving text-to-sign (T2S) as well as sign-to-text (S2T). We currently implement SignNet as a single channel architecture so that the output of the T2S task can be fed into S2T in a continuous dual learning framework. By single channel, we refer to a single modality, the body pose joints. In this work, we present SignNet, a T2S task using a novel metric embedding learning process, to preserve the distances between sign embeddings relative to their dissimilarity. We also describe how to choose positive and negative examples of signs for similarity testing. From our analysis, we observe that metric embedding learning-based model perform significantly better than the other models with traditional losses, when evaluated using BLEU scores. In the task of gloss to pose, SignNet performed as well as its state-of-the-art (SoTA) counterparts and outperformed them in the task of text to pose, by showing noteworthy enhancements in BLEU 1 - BLEU 4 scores (BLEU 1: 31 → 39; ≈26% improvement and BLEU 4: 10.43 →11.84; ≈14% improvement) when tested on the popular RWTH PHOENIX-Weather-2014T benchmark dataset
Adversarial Deep Multi-Task Learning Using Semantically Orthogonal Spaces and Application to Facial Attributes Prediction

Deep learning-based multi-task approaches usually rely on factorizing representation layers up to a certain point, where the network splits into several heads, each one addressing a specific task.Depending on the inter-task correlation, such naive model may or may not allow the tasks to benefit from each others.In this paper, we propose a novel Semantic Orthogonality Spaces (SOS) method for multi-task problems, where each task is predicted using the information from a common subspace that factorizes information among all tasks, as well as a task-specific subspace.We enforce orthogonality between these tasks by applying soft orthogonality constraints, as well as adversarially-learned semantic orthogonality objectives that ensures that predicting one task requires the specific information related to that task.We demonstrate the effectiveness of SOS on synthetic data, as well as for large-scale facial attributes prediction.In particular, we use SOS to craft a lightweight architecture that provides high-end accuracies on CelebA database.
Self-Supervised Face Presentation Attack Detection with Dynamic Grayscale Snippets

Face presentation attack detection (PAD) plays an important role in defending face recognition systems against presentation attacks. The success of PAD largely relies on supervised learning that requires a huge number of labeled data, which is especially challenging for videos and often requires expert knowledge. To avoid the costly collection of labeled data, this paper presents a novel method for self-supervised video representation learning via motion prediction. To achieve this, we exploit the temporal consistency based on three RGB frames which are acquired at three different times in the video sequence. The obtained frames are then transformed into grayscale images where each image is specified to three different channels such as R(red), G(green), and B(blue) to form a dynamic grayscale snippet (DGS). Motivated by this, the labels are automatically generated to increase the temporal diversity based on DGS by using the different temporal lengths of the videos, which prove to be very helpful for the downstream task. Benefiting from the self-supervised nature of our method, we report the results that outperform existing methods on four public benchmarks, namely, Replay-Attack, MSU-MFSD, CASIA-FASD, and OULU-NPU. Explainability analysis has been carried out through LIME and Grad-CAM techniques to visualize the most important features used in the DGS.
Controllable Facial Micro-element Synthesis using Segmentation Maps

In facial image synthesis, the style of the source image is converted using a reference image, or images with different styles are synthesized by each attribute using a facial attribute segmentation map. However, previous works cannot deal with the fine areas because the style is changed mostly in large areas such as hair, eyes, and mouth. To overcome the limitation, we propose a novel method of synthesizing a facial image with micro-level facial elements. A deep learning-based high-resolution image synthesis model is employed after generating a label image from the face RGB image through skin micro-element segmentation and face attribute segmentation. In the process of generating a label image for synthesizing skin micro-elements, we propose a technique for controlling skin micro-elements, enabling the generation of various label images from a single face label image. Throughout the proposed method, the areas of skin micro-elements can be edited and different skin types can be simulated. The experimental results show that the generated face is significantly improved by applying the proposed method. Moreover, various faces can be synthesized by changing the types and stages of skin micro-elements.
Localization using Multi-Focal Spatial Attention for Masked Face Recognition

Since the beginning of world-wide COVID-19 pandemic, facial masks have been recommended to limit the spread of the disease. However, these masks hide certain facial attributes. Hence, it has become difficult for existing face recognition systems to perform identity verification on masked faces. In this context, it is necessary to develop masked Face Recognition (MFR) for contactless biometric recognition systems. Thus, in this paper, we propose Complementary Attention Learning and Multi-Focal Spatial Attention that precisely removes masked region by training complementary spatial attention to focus on two distinct regions: masked regions and backgrounds. In our method, standard spatial attention and networks focus on unmasked regions, and extract mask-invariant features while minimizing the loss of the conventional Face Recognition (FR) performance. For conventional FR, we evaluate the performance on the IJB-C, Age-DB, CALFW, and CPLFW datasets. We evaluate the MFR performance on the ICCV2021-MFR/Insightface track, and demonstrate the improved performance on the both MFR and FR datasets. Additionally, we empirically verify that spatial attention of proposed method is more precisely activated in unmasked regions.
An efficient approach for real-time abnormal human behavior recognition on surveillance cameras

In recent years, abnormal human behavior recognition has become an attractive research topic of computer vision due to the rapid growth of demand to monitor human activities on closed-circuit television (CCTV) cameras. However, developing a deep learning-based model for abnormal/violent behavior recognition in surveillance systems is still quite challenging and costly due to inadequate data and model complexity. This paper presents an efficient approach to recognize violent behavior such as fighting, sexual harassment, and climbing fence in real-time on a multi-camera-one-edge-device system. Our approach develops a lightweight 3DCNN model trained by an effective optimization process to recognize human behavior from sequence frames of CCTV video signal input. In the optimization method, we utilize two advantages of deep learning techniques of knowledge distillation and contrastive learning to enhance the quality of the lightweight model on recognizing recorded human behaviors, which can help the student network learn distilled information from both the bigger model and contrastive object representations. We also establish a large CCTV human behavior video dataset containing 4,200 abnormal and 24,000 normal videos. The effectiveness of the proposed approach is shown by the high inference performance and impressive results evaluated on both public datasets the RWF-2000 dataset, the UCF101 dataset, and our collected datasets.
CoNFies: Controllable Neural Face Avatars

Neural Radiance Fields (NeRF) are compelling techniques for modeling dynamic 3D scenes from 2D image collections. These volumetric representations would be well suited for synthesizing novel facial expressions but for two problems. First, deformable NeRFs are object agnostic and model holistic movement of the scene: they can replay how the motion changes over time, but they cannot alter it in an interpretable way. Second, controllable volumetric representations typically require either time-consuming manual annotations or 3D supervision to provide semantic meaning to the scene. We propose a controllable neural representation for face self-portraits (CoNFies), that solves both of these problems within a common framework, and it can rely on automated processing. We use automated facial action recognition (AFAR) to characterize facial expressions as a combination of action units (AU) and their intensities. AUs provide both the semantic locations and control labels for the system. CoNFies outperformed competing methods for novel view and expression synthesis in terms of visual and anatomic fidelity of expressions.
Intrinsic Imaging Model Enhanced Contrastive Face Representation Learning

Humans can easily perceive numerous information from faces, only part of which has been achieved by a machine, thanks to the availability of large-scale face images with supervision signals of those specific tasks. More face perception tasks, like rare expression or attribute recognition, and genetic syndrome diagnosis, are not solved due to a critical shortage of supervised data. One possible way to solve these tasks is leveraging ubiquitous large-scale unsupervised face images and building a foundation face model via methods like contrastive learning (CL), which is, however, not aware of the intrinsic physics of the human face. In consideration of this shortcoming, this paper proposes to enhance contrastive face representation learning by the physical imaging model. Specifically, besides the CL-backbone network, we also design an auxiliary bypass pathway to constrain the CL-backbone to support the ability of accurately re-rendering the face with a differentiable physical imaging model after decomposing an input face image into intrinsic 3D imaging factors. With this design, the CL network is endowed the capacity of implicitly “knowing” the 3D of the face rather than the 2D pixels only. In experiments, we learn face representations from the CelebA and WebFace-42M datasets in unsupervised mode and evaluate the generalization capability of the representations with three different downstream tasks in case of limited supervised data. The experimental results clearly justify the effectiveness of the proposed method.
Practical Coded Sharding

Blockchain technology is finding applications not only in the cryptocurrency world but also in sectors like healthcare, IoT etc. However, before it can replace centralized counterparts like VISA or PayPal, its transaction throughput has to increase multiple orders of magnitude from its current throughput. Sharding is one solution for increasing the trans-action rate but it weakens the underlying security guarantees. Coded sharding uses information-theoretic approaches to scale blockchains without compromising security. Deploying coded sharding at webscale remains a challenge since it incurs bottlenecks like decoding very large Reed -Solomon codes of length of as much as 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">18</sup> . In this work, we propose a new method that leverages verifiable computing as an alternative to costly decoding strategies. We also simulate a simple balance checking application using our approach to check the feasibility of using verifiable computing with coded sharding. As its overheads are lower, we report transaction rates of 2806 tx/sec on commodity hardware. We also report the throughput of coded sharding based on verifiable computing by varying parameters such as the number of shards, users handled by each shard and complexity of verification used.
STIF: A Spatial–Temporal Integrated Framework for End-to-End Micro-UAV Trajectory Tracking and Prediction With 4-D MIMO Radar

The early trajectory prediction of micro unmanned aerial vehicles (micro-UAVs) with random behavior intentions facilitates the elimination of potential safety hazards. However, due to the property of a small radar cross Section (RCS), the backscattered radar signals from micro-UAVs may be submerged under strong background clutters, leading to distorted tracking and false prediction. To this end, this article presents a spatial–temporal integrated framework (STIF) for end-to-end micro-UAV trajectory tracking and prediction based on a 4-D multiple-input–multiple-output (MIMO) radar. Especially, to obtain accurate trajectories in low signal-to-noise ratio (SNR) conditions, the target detection and tracking are considered to be interdependent and addressed jointly in this work, rather than treating them as two separate processes in conventional methods. The advantage is that with the assistance of tracking, all consecutive spatial information encoded in raw radar streams can be incorporated to enhance the continuous detection performance, avoiding information loss using only one single scan. Subsequently, to accommodate high maneuvering scenarios, an intention-aware end-to-end transformer-based prediction framework is presented to simultaneously discover both spatial and temporal dependencies hiding in long-term estimated trajectories. Consequently, a 4-D frequency modulated continuous wave (FMCW) radar is utilized to evaluate the proposed system. Numerous simulation and experimental results indicate that STIF outperforms competing state-of-the-art methods and achieve superior prediction performance with the accuracy of 0.3851 m in low SNR conditions.
Text Growing on Leaf

Irregular-shaped texts bring challenges to Scene Text Detection (STD). Although existing regression-based approaches achieve comparable performances, they fail to cover some highly curved ribbon-like text lines. Inspired by morphology, we found that the leaf vein can easily cover various geometries. Specifically, lateral and thin veins are emitted to margin along main vein gradually with the leaf growth. This process can decompose a concave object into consecutive convex regions, which are easier to fit. Hence, the leaf vein is suitable for representing highly curved texts. Considering the aforementioned advantage, we design a leaf vein-based text representation method (LVT), where text contour is treated as leaf margin and represented through main, lateral, and thin veins. We further construct a detection framework based on LVT, namely LeafText. In the text reconstruction stage, LeafText simulates the leaf growth process to rebuild text contours. It grows main veins in Cartesian coordinates to locate texts roughly at first. Then, lateral and thin veins are generated along the main vein growth direction in polar coordinates. They are responsible for generating the coarse contour and refining it, respectively. Meanwhile, Multi-Oriented Smoother (MOS) is designed to smooth the main vein for ensuring reliable growth directions of lateral and thin veins. Additionally, a global incentive loss is proposed to enhance the predictions of lateral and thin veins. Ablation experiments demonstrate LVT can fit irregular-shaped texts precisely and verify the effectiveness of MOS and global incentive loss. Comparisons show that LeafText is superior to existing state-of-the-art (SOTA) methods on MSRA-TD500, CTW1500, Total-Text, and ICDAR2015 datasets.
Learning Dynamic Computing Resource Allocation in Convolutional Neural Networks for Wireless Interference Identification

Wireless interference identification (WII) is a critical technology for the non-cooperative communication systems, and it is widely applied into military and civilian scenarios. With enormous success in many computer vision and language processing applications, deep learning (DL) has also achieved the remarkable performance for WII. However, existing DL based WII methods can not dynamically allocate proper computing resources conditioned on the inputs, causing serious waste of resources and low computational efficiency, which is unacceptable for edge computing devices, such as unmanned aerial vehicle (UAV)-aided systems. In this paper, the problem of dynamic computation resource allocation is modeled as adaptively learning forward propagation depth of networks on a per-input basis, and this novel method is termed as dynamic computing resource allocation in convolutional neural networks (DCRACNN). In particular, the proposed DCRACNN is composed of two subnetworks: a main network and a policy network. The main network is equipped with multiple classifiers at different depths, allowing test examples to stop early during inference so as to significantly improve computational efficiency. The policy network is optimized with reinforcement learning to address the bottleneck of non-differentiable decisions of optimal forward propagation depth for the main network while preserving recognition accuracy. To address the challenges of high error rates of early classifiers in the main network, novel methods are introduced, which can be seamlessly integrated into DRCACNN. In addition, a novel network pruning process is proposed for the main network to further reduce model sizes and computational complexity. Experiments demonstrate that DCRACNN can significantly reduce the calculation cost with boosting accuracy for WII.
A Hybrid Approach to 3D Shape Estimation of Catheters Using Ultrasound Images

Catheter-based cardiac interventions have substantially improved the efficacy of the treatment. However, catheter navigation still poses several challenges including its poor visualization and localisation. Therefore, various imaging methods have been used to localize the catheter and estimate its shape. The use of ultrasound imaging is superior to other modalities in many aspects, but suffers from poor spatial resolution. Hence, we present a hybrid approach involving deep learning and classical approach to 3D shape estimation of catheters. Our novel approach uses two stages. First, a UNet-3D model is proposed to estimate the catheter centroid in the ultrasound image. Then, an adaptive Kalman Filter (AKF) is used to fuse the points into the 3D world coordinate frame. Simulation studies, phantom and ex-vivo experimentation results demonstrate the robustness of the method to ultrasound noise and extreme configurations (sharp curves).
A Prediction Based Replica Selection Strategy for Reducing Tail Latency in Geo-Distributed Systems

The deployment of modern applications in geo-distributed systems results in performance fluctuation which is a consequence of long-tail latency. To deliver high-quality services these applications always strive to adapt to the changing situation and an appropriate replica selection strategy is one efficient way to achieve this. Several replica selection strategies have already been developed but none of them are efficient enough to reduce tail latency and adapt to the dynamic environment of the geo-distributed systems. In this paper, we present the design and implementation of a prediction based replica selection strategy for reducing tail latency in geo-distributed systems. We have meticulously designed the proposed strategy to adapt to the dynamic behavior of the distributed system. For evaluating its effectiveness in reducing tail latency and adapting to the dynamic behavior of the geo-distributed systems we perform some extensive experiments in a 15 nodes Cassandra cluster that is deployed on Amazon EC2 over 5 geographical regions. For generating test datasets and workloads we use industry-standard Yahoo Cloud Serving Benchmark (YCSB). Our experimental results show that the proposed strategy not only reduces tail latency but also increases the overall throughput of the systems.