Reviewerly/exHarmony huggingface dataset:
DarKnight: An Accelerated Framework for Privacy and Integrity Preserving Deep Learning Using Trusted Hardware

Privacy and security-related concerns are growing as machine learning reaches diverse application domains. The data holders want to train or infer with private data while exploiting accelerators, such as GPUs, that are hosted in the cloud. Cloud systems are vulnerable to attackers that compromise the privacy of data and integrity of computations. Tackling such a challenge requires unifying theoretical privacy algorithms with hardware security capabilities. This paper presents DarKnight, a framework for large DNN training while protecting input privacy and computation integrity. DarKnight relies on cooperative execution between trusted execution environments (TEE) and accelerators, where the TEE provides privacy and integrity verification, while accelerators perform the bulk of the linear algebraic computation to optimize the performance. In particular, DarKnight uses a customized data encoding strategy based on matrix masking to create input obfuscation within a TEE. The obfuscated data is then offloaded to GPUs for fast linear algebraic computation. DarKnight’s data obfuscation strategy provides provable data privacy and computation integrity in the cloud servers. While prior works tackle inference privacy and cannot be utilized for training, DarKnight’s encoding scheme is designed to support both training and inference.
Self-supervised Multi-view Multi-Human Association and Tracking

Multi-view Multi-human association and tracking (MvMHAT) aims to track a group of people over time in each view, as well as to identify the same person across different views at the same time. This is a relatively new problem but is very important for multi-person scene video surveillance. Different from previous multiple object tracking (MOT) and multi-target multi-camera tracking (MTMCT) tasks, which only consider the over-time human association, MvMHAT requires to jointly achieve both cross-view and over-time data association. In this paper, we model this problem with a self-supervised learning framework and leverage an end-to-end network to tackle it. Specifically, we propose a spatial-temporal association network with two designed self-supervised learning losses, including a symmetric-similarity loss and a transitive-similarity loss, at each time to associate the multiple humans over time and across views. Besides, to promote the research on MvMHAT, we build a new large-scale benchmark for the training and testing of different algorithms. Extensive experiments on the proposed benchmark verify the effectiveness of our method. We have released the benchmark and code to the public.
M3TR: Multi-modal Multi-label Recognition with Transformer

Multi-label image recognition aims to recognize multiple objects simultaneously in one image. Recent ideas to solve this problem have focused on learning dependencies of label co-occurrences to enhance the high-level semantic representations. However, these methods usually neglect the important relations of intrinsic visual structures and face difficulties in understanding contextual relationships. To build the global scope of visual context as well as interactions between visual modality and linguistic modality, we propose the Multi-Modal Multi-label recognition TRansformers (M3TR) with the ternary relationship learning for inter-and intra-modalities. For the intra-modal relationship, we make insightful conjunction of CNNs and Transformers, which embeds visual structures into the high-level features by learning the semantic cross-attention. For constructing the interactions between the visual and linguistic modalities, we propose a linguistic cross-attention to embed the class-wise linguistic information into the visual structure learning, and finally present a linguistic guided enhancement module to enhance the representation of high-level semantics. Experimental evidence reveals that with the collaborative learning of ternary relationship, our proposed M3TR achieves new state-of-the-art on two public multi-label recognition benchmarks.
Self-Supervised Speaker Recognition with Loss-Gated Learning

In self-supervised learning for speaker recognition, pseudo labels are useful as the supervision signals. It is a known fact that a speaker recognition model doesn’t always benefit from pseudo labels due to their unreliability. In this work, we observe that a speaker recognition network tends to model the data with reliable labels faster than those with unreliable labels. This motivates us to study a loss-gated learning (LGL) strategy, which extracts the reliable labels through the fitting ability of the neural network during training. With the proposed LGL, our speaker recognition model obtains a 46.3% performance gain over the system without it. Further, the proposed self-supervised speaker recognition with LGL trained on the VoxCeleb2 dataset without any labels achieves an equal error rate of 1.66% on the VoxCeleb1 original test set.
LSSNet

Macro- and micro-expression spotting is a very challenging task to locate their occurrence intervals in long face videos. In this paper, we propose an efficient two-stream network named location suppression based spotting network (LSSNet), which includes three parts. First, the optical flow is extracted using the traditional TV-L1 algorithm which captures subtle facial movements while adding temporal information to alleviate the problem of insufficient samples. Then, fixed length features are extracted from the sampled optical flow and raw images by an I3D model, which is used to set sliding windows. Finally, location suppression modules (LSMs) are added to the pyramidal convolutional neural network (CNN) to reduce the proposals with too long and too short intervals. In addition, we use two different methods, named top_k and top_threshold, for validation. We adopt leave-one-subject-out (LOSO) to train our model on CAS(ME)2 and SAMM-LV. Experimental results show that our LSSNet achieves the state-of-the-art result with top_threshold, especially on the CAS(ME)2 dataset. The code is available at https://github.com/williamlee91/mer_spot.
Legitimate Adversarial Patches

It is known that deep neural models are vulnerable to adversarial attacks. Digital attacks can craft imperceptible perturbations but lack of the ability to apply in physical environment. To address this issue, efforts have been investigated to study physical patch attacks in the physical world, especially for object detection models. Previous works mostly focus on evading the detection model itself but ignore the impact of human observers. In this paper, we study legitimate adversarial attacks that evade both human eyes and detection models in the physical world. To this end, we delve into the issue of patch rationality, and propose some indicators for evaluating the rationality of physical adversarial patches. Besides, we propose a novel framework with a two-stage training strategy to generate our legitimate adversarial patches (LAPs). Both in numerical simulations and physical experiments our LAPs have significant attack effects and visual rationality.
SOGAN

In recent years, virtual makeup applications have become more and more popular. However, it is still challenging to propose a robust makeup transfer method in the real-world environment. Current makeup transfer methods mostly work well on good-conditioned clean makeup images, but transferring makeup that exhibits shadow and occlusion is not satisfying. To alleviate it, we propose a novel makeup transfer method, called 3D-Aware Shadow and Occlusion Robust GAN (SOGAN). Given the source and the reference faces, we first fit a 3D face model and then disentangle the faces into shape and texture. In the texture branch, we map the texture to the UV space and design a UV texture generator to transfer the makeup. Since human faces are symmetrical in the UV space, we can conveniently remove the undesired shadow and occlusion from the reference image by carefully designing a Flip Attention Module (FAM). After obtaining cleaner makeup features from the reference image, a Makeup Transfer Module (MTM) is introduced to perform accurate makeup transfer. The qualitative and quantitative experiments demonstrate that our SOGAN not only achieves superior results in shadow and occlusion situations but also performs well in large pose and expression variations.
Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reduction

A number of deep learning based algorithms have been proposed to recover high-quality videos from low-quality compressed ones. Among them, some restore the missing details of each frame via exploring the spatiotemporal information of neighboring frames. However, these methods usually suffer from a narrow temporal scope, thus may miss some useful details from some frames outside the neighboring ones. In this paper, to boost artifact removal, on the one hand, we propose a Recursive Fusion (RF) module to model the temporal dependency within a long temporal range. Specifically, RF utilizes both the current reference frames and the preceding hidden state to conduct better spatiotemporal compensation. On the other hand, we design an efficient and effective Deformable Spatiotemporal Attention (DSTA) module such that the model can pay more effort on restoring the artifact-rich areas like the boundary area of a moving object. Extensive experiments show that our method outperforms the existing ones on the MFQE 2.0 dataset in terms of both fidelity and perceptual effect. Code is available at https://github.com/zhaominyiz/RFDA-PyTorch.
M<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si1.svg"><mml:msup><mml:mrow /><mml:mn>3</mml:mn></mml:msup></mml:math>Net: A multi-scale multi-view framework for multi-phase pancreas segmentation based on cross-phase non-local attention

The complementation of arterial and venous phases visual information of CTs can help better distinguish the pancreas from its surrounding structures. However, the exploration of cross-phase contextual information is still under research in computer-aided pancreas segmentation. This paper presents M3Net, a framework that integrates multi-scale multi-view information for multi-phase pancreas segmentation. The core of M3Net is built upon a dual-path network in which individual branches are set up for two phases. Cross-phase interactive connections bridging the two branches are introduced to interleave and integrate dual-phase complementary visual information. Besides, we further devise two types of non-local attention modules to enhance the high-level feature representation across phases. First, we design a location attention module to generate cross-phase reliable feature correlations to suppress the misalignment regions. Second, the depth-wise attention module is used to capture the channel dependencies and then strengthen feature representations. The experiment data consists of 224 internal CTs (106 normal and 118 abnormal) with 1 mm slice thickness, and 66 external CTs (29 normal and 37 abnormal) with 5 mm slice thickness. We achieve new state-of-the-art performance with average DSC of 91.19% on internal data, and promising result with average DSC of 86.34% on external data.
TitaNet: Neural Model for Speaker Representation with 1D Depth-Wise Separable Convolutions and Global Context

In this paper, we propose TitaNet, a novel neural network architecture for extracting speaker representations. We employ 1D depth-wise separable convolutions with Squeeze-and-Excitation (SE) layers with global context followed by channel attention based statistics pooling layer to map variable-length utterances to a fixed-length embedding (t-vector). TitaNet is a scalable architecture and achieves state-of-the-art performance on speaker verification task with an equal error rate (EER) of 0.68% on the VoxCeleb1 trial file and also on speaker diarization tasks with diarization error rate (DER) of 1.73% on AMI-MixHeadset, 1.99% on AMI-Lapel and 1.11% on CH109. Furthermore, we investigate various sizes of TitaNet and present a light TitaNet-S model with only 6M parameters that achieve near state-of-the-art results in diarization tasks.
Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction

In the present paper, we propose a decoder-free extension of Dreamer, a leading model-based reinforcement learning (MBRL) method from pixels. Dreamer is a sample- and cost-efficient solution to robot learning, as it is used to train latent state-space models based on a variational autoencoder and to conduct policy optimization by latent trajectory imagination. However, this autoencoding based approach often causes object vanishing, in which the autoencoder fails to perceives key objects for solving control tasks, and thus significantly limiting Dreamer's potential. This work aims to relieve this Dreamer's bottleneck and enhance its performance by means of removing the decoder. For this purpose, we firstly derive a likelihood- free and InfoMax objective of contrastive learning from the evidence lower bound of Dreamer. Secondly, we incorporate two components, (i) independent linear dynamics and (ii) the random crop data augmentation, to the learning scheme so as to improve the training performance. In comparison to Dreamer and other recent model-free reinforcement learning methods, our newly devised Dreamer with InfoMax and without generative decoder (Dreaming) achieves the best scores on 5 difficult simulated robotics tasks, in which Dreamer suffers from object vanishing.
Educational impact of an Augmented Reality (AR) application for teaching structural systems to non-engineering students

This study evaluates the effectiveness of Augmented Reality (AR) in improving undergraduate non-engineering students’ (n = 103) achievement of learning outcomes and their perception of AR as a learning tool. An AR application for teaching structural systems, Virtual and Augmented Reality for Structures (VARS), is developed and tested using a quasi-experiment and a questionnaire. The study shows that VARS AR significantly improves the posttest quiz score of the experimental group. In addition, a conceptual framework linking design considerations and achievement of learning outcomes is created. Based on the questionnaire survey, authentic context and user interface are important design considerations influencing students’ perceived achievement of learning outcomes. It was found that experience using AR and background knowledge did not influence the perceived achievement of learning outcomes and change in quiz score.
Multi-Sensor Filtering Fusion With Parametric Uncertainties and Measurement Censoring: Monotonicity and Boundedness

This paper is concerned with the Tobit Kalman fusion estimation problem for a class of multi-sensor systems subject to parametric uncertainties and measurement censoring. The parametric uncertainty is characterized by the multiplicative noise appearing in both state and observation equations, and the measurement censoring is governed by the Tobit observation model. The fusion estimation is implemented via two stages: at the first stage, each sensor sends its observations to the local estimator and, at the second stage, the local estimates are then transmitted to the fusion center so as to generate the fused estimate. The local estimator realizes a Tobit Kalman filtering algorithm which is devised in accordance with a modified regression model, whilst the fusion center carries out the fusion estimation by resorting to the federated fusion rule. Furthermore, the monotonicity of the fused error covariance with respect to the censoring threshold is discussed and, subsequently, the boundedness property is also examined where both lower and upper bounds are acquired for the fused error covariance. The validity of the fusion estimator is finally shown via two numerical examples.
How reliable is smartphone-based electronic contact tracing for COVID-19?

Smartphone-based electronic contact tracing is currently considered an essential tool towards easing lockdowns, curfews, and shelter-in-place orders issued by most governments around the world in response to the 2020 novel coronavirus (SARS-CoV-2) crisis. While the focus on developing smartphone-based contact tracing applications or apps has been on privacy concerns stemming from the use of such apps, an important question that has not received sufficient attention is: How reliable will such smartphone-based electronic contact tracing be? This is a technical question related to how two smartphones reliably register their mutual proximity. Here, we examine in detail the technical prerequisites required for effective smartphone-based contact tracing. The underlying mechanism that any contact tracing app relies on is called Neighbor Discovery (ND), which involves smartphones transmitting and scanning for Bluetooth signals to record their mutual presence whenever they are in close proximity. The hardware support and the software protocols used for ND in smartphones, however, were not designed for reliable contact tracing. In this paper, we quantitatively evaluate how reliably can smartphones do contact tracing. Our results point towards the design of a wearable solution for contact tracing that can overcome the shortcomings of a smartphone-based solution to provide more reliable and accurate contact tracing. To the best of our knowledge, this is the first study that quantifies, both, the suitability and also the drawbacks of smartphone-based contact tracing. Further, our results can be used to parameterize a ND protocol to maximize the reliability of any contact tracing app that uses it.
Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning

Making decisions in complex driving environments is a challenging task for autonomous agents. Imitation learning methods have great potentials for achieving such a goal. Adversarial Inverse Reinforcement Learning (AIRL) is one of the state-of-art imitation learning methods that can learn both a behavioral policy and a reward function simultaneously, yet it is only demonstrated in simple and static environments where no interactions are introduced. In this paper, we improve and stabilize AIRL&#x2019;s performance by augmenting it with semantic rewards in the learning framework. Additionally, we adapt the augmented AIRL to a more practical and challenging decision-making task in a highly interactive environment in autonomous driving. The proposed method is compared with four baselines and evaluated by four performance metrics. Simulation results show that the augmented AIRL outperforms all the baseline methods, and its performance is comparable with that of the experts on all of the four metrics.
Arbitrary optical wave evolution with Fourier transforms and phase masks

A large number of applications in classical and quantum photonics require the capability of implementing arbitrary linear unitary transformations on a set of optical modes. In a seminal work by Reck et al. [Phys. Rev. Lett.73, 58 (1994)10.1103/PhysRevLett.73.58], it was shown how to build such multiport universal interferometers with a mesh of beam splitters and phase shifters, and this design became the basis for most experimental implementations in the last decades. However, the design of Reck et al. is difficult to scale up to a large number of modes, which would be required for many applications. Here we present a deterministic algorithm that can find an exact and efficient implementation of any unitary transformation, using only Fourier transforms and phase masks. Since Fourier transforms and phase masks are routinely implemented in several optical setups and they do not suffer from some of the scalability issues associated with building extensive meshes of beam splitters, we believe that our design can be useful for many applications in photonics.
SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation

This paper focuses on visual semantic navigation, the task of producing actions for an active agent to navigate to a specified target object category in an unknown environment. To complete this task, the algorithm should simultaneously locate and navigate to an instance of the category. In comparison to the traditional point goal navigation, this task requires the agent to have a stronger contextual prior to indoor environments. We introduce SSCNav, an algorithm that explicitly models scene priors using a confidence-aware semantic scene completion module to complete the scene and guide the agent's navigation planning. Given a partial observation of the environment, SSC-Nav first infers a complete scene representation with semantic labels for the unobserved scene together with a confidence map associated with its own prediction. Then, a policy network infers the action from the scene completion result and confidence map. Our experiments demonstrate that the proposed scene completion module improves the efficiency of the downstream navigation policies. Code and data: https://sscnav.cs.columbia.edu/
Vision-based Crowd Counting and Social Distancing Monitoring using Tiny-YOLOv4 and DeepSORT

With the novel coronavirus, social distancing and crowd monitoring became vital in managing the spread of the virus. This paper presents a desktop application that utilizes Tiny-YOLOv4 and DeepSORT tracking algorithm to monitor crowd count and social distancing in a top-view camera perspective. The application is able to process video files or live camera feed such as CCTV or surveillance cameras and generate reports indicating people detected per unit time, percentage of social distancing per unit time, detection and social distancing logs as well as color-coded bounding boxes to indicate if the detected people are following social distancing protocols.
Progressive Graph Attention Network for Video Question Answering

Video question answering~(Video-QA) is a task of answering a natural language question related to the content of a video. Existing methods generally explore the single interactions between objects or between frames, which are insufficient to deal with the sophisticated scenes in videos. To tackle this problem, we propose a novel model, termed Progressive Graph Attention Network (PGAT), which can jointly explore the multiple visual relations on object-level, frame-level and clip-level. Specifically, in the object-level relation encoding, we design two kinds of complementary graphs, one for learning the spatial and semantic relations between objects from the same frame, the other for modeling the temporal relations between the same object from different frames. The frame-level graph explores the interactions between diverse frames to record the fine-grained appearance change, while the clip-level graph models the temporal and semantic relations between various actions from clips. These different-level graphs are concatenated in a progressive manner to learn the visual relations from low-level to high-level. Furthermore, we for the first time identified that there are serious answer biases with TGIF-QA, a very large Video-QA dataset, and reconstructed a new dataset based on it to overcome the biases, called TGIF-QA-R. We evaluate the proposed model on three benchmark datasets and the new TGIF-QA-R, and the experimental results demonstrate that our model significantly outperforms other state-of-the-art models. Our codes and dataset are available at https://github.com/PengLiang-cn/PGAT.