Reviewerly/exHarmony huggingface dataset:
PocketNet: Extreme Lightweight Face Recognition Network Using Neural Architecture Search and Multistep Knowledge Distillation

Deep neural networks have rapidly become the mainstream method for face recognition (FR). However, this limits the deployment of such models that contain an extremely large number of parameters to embedded and low-end devices. In this work, we present an extremely lightweight and accurate FR solution, namely PocketNet. We utilize neural architecture search to develop a new family of lightweight face-specific architectures. We additionally propose a novel training paradigm based on knowledge distillation (KD), the multi-step KD, where the knowledge is distilled from the teacher model to the student model at different stages of the training maturity. We conduct a detailed ablation study proving both, the sanity of using NAS for the specific task of FR rather than general object classification, and the benefits of our proposed multi-step KD. We present an extensive experimental evaluation and comparisons with the state-of-the-art (SOTA) compact FR models on nine different benchmarks including large-scale evaluation benchmarks such as IJB-C and MegaFace. PocketNets have consistently advanced the SOTA FR performance on nine mainstream benchmarks when considering the same level of model compactness. With 0.92M parameters, our smallest network PocketNetS-128 achieved very competitive results to recent SOTA compacted models that contain up to 4M parameters. Training codes and pre-trained models are public. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><sup>1</sup></xref>
Discrete online cross-modal hashing

With the prevalence of multimedia content on the Web which usually continuously comes in a stream fashion, online cross-modal hashing methods have attracted extensive interest in recent years. However, most online hashing methods adopt a relaxation strategy or real-valued auxiliary variable strategy to avoid complex optimization of hash codes, leading to large quantization errors. In this paper, based on Discrete Latent Factor model-based cross-modal Hashing (DLFH), we propose a novel cross-modal online hashing method, i.e., Discrete Online Cross-modal Hashing (DOCH). To generate uniform high-quality hash codes of different modal, DOCH not only directly exploits the similarity between newly coming data and old existing data in the Hamming space, but also utilizes the fine-grained semantic information by label embedding. Moreover, DOCH can discretely learn hash codes by an efficient optimization algorithm. Extensive experiments conducted on two real-world datasets demonstrate the superiority of DOCH.
Self Supervision to Distillation for Long-Tailed Visual Recognition

Deep learning has achieved remarkable progress for visual recognition on large-scale balanced datasets but still performs poorly on real-world long-tailed data. Previous methods often adopt class re-balanced training strategies to effectively alleviate the imbalance issue, but might be a risk of over-fitting tail classes. The recent decoupling method overcomes over-fitting issues by using a multi-stage training scheme, yet, it is still incapable of capturing tail class information in the feature learning stage. In this paper, we show that soft label can serve as a powerful solution to incorporate label correlation into a multi-stage training scheme for long-tailed recognition. The intrinsic relation between classes embodied by soft labels turns out to be helpful for long-tailed recognition by transferring knowledge from head to tail classes.Specifically, we propose a conceptually simple yet particularly effective multi-stage training scheme, termed as Self Supervised to Distillation (SSD). This scheme is composed of two parts. First, we introduce a self-distillation framework for long-tailed recognition, which can mine the label relation automatically. Second, we present a new distillation label generation module guided by self-supervision. The distilled labels integrate information from both label and data domains that can model long-tailed distribution effectively. We conduct extensive experiments and our method achieves the state-of-the-art results on three long-tailed recognition benchmarks: ImageNet-LT, CIFAR100-LT and iNaturalist 2018. Our SSD outperforms the strong LWS baseline by from 2.7% to 4.5% on various datasets.
Fully Complex-Valued Dendritic Neuron Model

A single dendritic neuron model (DNM) that owns the nonlinear information processing ability of dendrites has been widely used for classification and prediction. Complex-valued neural networks that consist of a number of multiple/deep-layer McCulloch-Pitts neurons have achieved great successes so far since neural computing was utilized for signal processing. Yet no complex value representations appear in single neuron architectures. In this article, we first extend DNM from a real-value domain to a complex-valued one. Performance of complex-valued DNM (CDNM) is evaluated through a complex XOR problem, a non-minimum phase equalization problem, and a real-world wind prediction task. Also, a comparative analysis on a set of elementary transcendental functions as an activation function is implemented and preparatory experiments are carried out for determining hyperparameters. The experimental results indicate that the proposed CDNM significantly outperforms real-valued DNM, complex-valued multi-layer perceptron, and other complex-valued neuron models.
Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation

3D human shape and pose estimation is the essential task for human motion analysis, which is widely used in many 3D applications. However, existing methods cannot simultaneously capture the relations at multiple levels, including spatial-temporal level and human joint level. Therefore they fail to make accurate predictions in some hard scenarios when there is cluttered background, occlusion, or extreme pose. To this end, we propose Multi-level Attention Encoder-Decoder Network (MAED), including a Spatial-Temporal Encoder (STE) and a Kinematic Topology Decoder (KTD) to model multi-level attentions in a unified framework. STE consists of a series of cascaded blocks based on Multi-Head Self-Attention, and each block uses two parallel branches to learn spatial and temporal attention respectively. Meanwhile, KTD aims at modeling the joint level attention. It regards pose estimation as a top-down hierarchical process similar to SMPL kinematic tree. With the training set of 3DPW, MAED outperforms previous state-of-the-art methods by 6.2, 7.2, and 2.4 mm of PA-MPJPE on the three widely used benchmarks 3DPW, MPI-INF-3DHP, and Human3.6M respectively. Our code is available at https://github.com/ziniuwan/maed.
Preservational Learning Improves Self-supervised Medical Image Models by Reconstructing Diverse Contexts

Preserving maximal information is one of principles of designing self-supervised learning methodologies. To reach this goal, contrastive learning adopts an implicit way which is contrasting image pairs. However, we believe it is not fully optimal to simply use the contrastive estimation for preservation. Moreover, it is necessary and complemental to introduce an explicit solution to preserve more information. From this perspective, we introduce Preservational Learning to reconstruct diverse image contexts in order to preserve more information in learned representations. Together with the contrastive loss, we present Preservational Contrastive Representation Learning (PCRL) for learning self-supervised medical representations. PCRL provides very competitive results under the pretraining-finetuning protocol, outperforming both self-supervised and supervised counterparts in 5 classification/segmentation tasks substantially. Codes are available at https://github.com/Luchixiang/PCRL.
Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer

Designing hardware accelerators for deep neural networks (DNNs) has been much desired. Nonetheless, most of these existing accelerators are built for either convolutional neural networks (CNNs) or recurrent neural networks (RNNs). Recently, the Transformer model is replacing the RNN in the natural language processing (NLP) area. However, because of intensive matrix computations and complicated data flow being involved, the hardware design for the Transformer model has never been reported. In this paper, we propose the first hardware accelerator for two key components, i.e., the multi-head attention (MHA) ResBlock and the position-wise feed-forward network (FFN) ResBlock, which are the two most complex layers in the Transformer. Firstly, an efficient method is introduced to partition the huge matrices in the Transformer, allowing the two ResBlocks to share most of the hardware resources. Secondly, the computation flow is well designed to ensure the high hardware utilization of the systolic array, which is the biggest module in our design. Thirdly, complicated nonlinear functions are highly optimized to further reduce the hardware complexity and also the latency of the entire system. Our design is coded using hardware description language (HDL) and evaluated on a Xilinx FPGA. Compared with the implementation on GPU with the same setting, the proposed design demonstrates a speed-up of 14.6 x in the MHA ResBlock, and 3.4 x in the FFN ResBlock, respectively. Therefore, this work lays a good foundation for building efficient hardware accelerators for multiple Transformer networks.
Direct (3+1)D laser writing of graded-index optical elements

We propose single-step additive fabrication of graded-index optical elements by introducing the light exposure as the additional dimension to three-dimensional (3D) laser writing, hence (3+1)D writing. We use a commercial printer and photoresist to realize the proposed single step fabrication method that can be swiftly adopted for research and engineering. After presenting the characterization of the graded-index profiles via basic structures, we demonstrate two different optical devices: volume holograms that are superimposed using angular and peristrophic multiplexing, and optical waveguides with well-defined refractive index profiles. In the latter, we precisely control the propagating modes via tuning the (3+1)D-printed waveguide parameters and report step-index and graded-index core-cladding transitions.
Security, Privacy, and Usability in Continuous Authentication: A Survey

Continuous authentication has been proposed as a possible approach for passive and seamless user authentication, using sensor data comprising biometric, behavioral, and context-oriented characteristics. Since these are personal data being transmitted and are outside the control of the user, this approach causes privacy issues. Continuous authentication has security challenges concerning poor matching rates and susceptibility of replay attacks. The security issues are mainly poor matching rates and the problems of replay attacks. In this survey, we present an overview of continuous authentication and comprehensively discusses its different modes, and issues that these modes have related to security, privacy, and usability. A comparison of privacy-preserving approaches dealing with the privacy issues is provided, and lastly recommendations for secure, privacy-preserving, and user-friendly continuous authentication.
Enhanced Marine Predators Algorithm with Local Escaping Operator for Global Optimization

The recently introduced Marine Predators Algorithm (MPA) exhibits competitive performance in solving optimization problems. However, it often prematurely converges due to an imbalance between its exploration and exploitation capabilities. Therefore, in this paper, an improved MPA variant using a proposed Local Escaping Operator (LEO) is introduced. In the MPA, solution candidates are replaced by better candidates from the previous iteration, mimicking a memory of already visited prey-abundant areas. This indicates a weak inter-dependence between the candidates in the population and possible acceptance of new solution candidates created outside the algorithm without damaging the optimization process. Consequently, in the proposed approach, the worst candidates are replaced with solutions created by the LEO. The LEO is based on representative solutions, taking into account the relationship between predators and the characteristics of their population. The approach is experimentally compared with the state-of-the-art meta-heuristics on 82 test functions, including IEEE Congress on Evolutionary Computation test suites (CEC’14 and CEC’17) and three engineering problems. The results show the superiority of the LEO-MPA over the MPA and recent algorithms. Furthermore, in this paper, the suitability of the hybridization of meta-heuristics with the LEO is discussed and successful attempts with the best algorithms are shown.
“Nice to see you virtually”: Thoughtful design and evaluation of virtual avatar of the other user in AR and VR based telexistence systems

This paper presents two studies investigating how physically remote telexistence users wish to see other users visualized as virtual avatars in a) augmented reality, and b) immersive virtual reality while conducting a collaborative task. To answer this research question, a telexistence system was designed and implemented with simple avatar designs. After that, visual examples of alternative avatar representations for both use cases were designed by thoughtfully altering the visual parameters of 36 virtual avatar examples. The avatar designs were first evaluated in a user study with 16 participants in conjunction with using an implemented telexistence system. As a follow-up an online survey with 43 respondents was used to record their preferences regarding virtual avatar appearance. The results suggest that users prefer the other user to be represented in a photorealistic full-body human avatar in both augmented reality and virtual reality due to its humanlike representation and affordances for interaction. In augmented reality, the choice for a hologram full body avatar was also popular due to its see-through appearance, which prevents a mix-up with a real person in the physical space.
Meta-heuristic as manager in federated learning approaches for image processing purposes

The new form of artificial intelligence training, i.e. federated learning, is becoming more popular in the last few years. It is an optimization problem that includes additional mechanisms such as aggregation and data transmission. In this paper, we propose a hybridization of this type of training with a meta-heuristic. The meta-heuristic algorithm is adapted to manage the entire process as well as to analyze the best models to minimize attacks on this type of collaboration. The proposed solution is based on minimizing the general model error, with additional control mechanisms for incoming models, or adapting the aggregation method depending on the quality of the model. The innovative solution has been analyzed in terms of its application to the problem of image classification using classical and convolutional neural networks, and the most popular meta-heuristic algorithms. The proposal was analyzed in terms of the accuracy of the general model as well as for security against poisoning attacks. We reached 91% of accuracy using the proposed method with the Red Fox Optimization Algorithm and 95% in terms of detection of poisoned samples in the database.
BDFL: A Byzantine-Fault-Tolerance Decentralized Federated Learning Method for Autonomous Vehicle

Autonomous Vehicles ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$AV$</tex-math></inline-formula> s) take advantage of Machine Learning (ML) for yielding improved experiences of self-driving. However, large-scale collection of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$AV$</tex-math></inline-formula> s’ data for training will inevitably result in a privacy leakage problem. Federated Learning (FL) is proposed to solve privacy leakage problems, but it is exposed to security threats such as model inversion, membership inference. Therefore, the vulnerability of the FL should be brought to the forefront when applying to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$AV$</tex-math></inline-formula> s. We propose a novel Byzantine-Fault-Tolerant (BFT) decentralized FL method with privacy-preservation for <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$AV$</tex-math></inline-formula> s called BDFL. In this paper, a Peer-to-Peer (P2P) FL with BFT is built by extending the HydRand protocol. In order to protect their model, each <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$AV$</tex-math></inline-formula> uses the Publicly Verifiable Secret Sharing(PVSS) scheme, which allows anyone to verify the correctness of encrypted shares. The evaluation results on the MNIST dataset have shown that introducing decentralized FL into <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ AV$</tex-math></inline-formula> area is feasible, and the proposed BDFL is superior to other BFT-based FL method. Furthermore, the experimental results on KITTI dataset indicate the practicality of BDFL on improving performances of multi-object recognition in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ AV$</tex-math></inline-formula> areas. Finally, the proposed PVSS-based data privacy preservation scheme is also justified its characteristic of no side-effect on models’ parameters by the experiments on the MNIST and KITTI datasets.
Leveraging blockchain for ensuring trust in IoT: A survey

Internet of Things (IoT) refers to a technology where computing devices are connected and form a network. IoT faces many security and privacy issues due to less computation power, heterogeneity, and limited resources available with its devices. Data is transferred among these devices with little or no human interaction. Data Confidentiality and Integrity are very critical parameters and can be achieved by securely sharing information in IoT scenarios. Managing and maintaining trust in exchanging information over IoT becomes very significant. Recent researches have focused on the applications of Blockchain technology for assuring trust management in IoT networks. Blockchain provides completely distinct and more secure approaches. This survey paper aims to illustrate the significance of integrating Blockchain technology in the IoT environment to ensure trust among IoT devices. Particularly, first we give an overview and security aspects of IoT and Blockchain technologies. Then we trace out some important challenges and issues of trusted IoT with potential solutions by Blockchain. Following this, we highlight some complications in the integration of Blockchain with IoT. Finally, we present a comparative analysis between traditional and Blockchain-based trust management techniques as proof of work to represent the significance of Blockchain in ensuring trust.
Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer

Encoder-decoder models have made great progress on handwritten mathematical expression recognition recently. However, it is still a challenge for existing methods to assign attention to image features accurately. Moreover, those encoder-decoder models usually adopt RNN-based models in their decoder part, which makes them inefficient in processing long Open image in new window sequences. In this paper, a transformer-based decoder is employed to replace RNN-based ones, which makes the whole model architecture very concise. Furthermore, a novel training strategy is introduced to fully exploit the potential of the transformer in bidirectional language modeling. Compared to several methods that do not use data augmentation, experiments demonstrate that our model improves the ExpRate of current state-of-the-art methods on CROHME 2014 by 2.23%. Similarly, on CROHME 2016 and CROHME 2019, we improve the ExpRate by 1.92% and 2.28% respectively.
Efficient <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si2.svg"><mml:mi>k</mml:mi></mml:math>-nearest neighbor search based on clustering and adaptive <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si2.svg"><mml:mi>k</mml:mi></mml:math> values

The k-Nearest Neighbor (kNN) algorithm is widely used in the supervised learning field and, particularly, in search and classification tasks, owing to its simplicity, competitive performance, and good statistical properties. However, its inherent inefficiency prevents its use in most modern applications due to the vast amount of data that the current technological evolution generates, being thus the optimization of kNN-based search strategies of particular interest. This paper introduces the caKD+ algorithm, which tackles this limitation by combining the use of feature learning techniques, clustering methods, adaptive search parameters per cluster, and the use of pre-calculated K-Dimensional Tree structures, and results in a highly efficient search method. This proposal has been evaluated using 10 datasets and the results show that caKD+ significantly outperforms 16 state-of-the-art efficient search methods while still depicting such an accurate performance as the one by the exhaustive kNN search.
Context-aware Scene Graph Generation with Seq2Seq Transformers

Scene graph generation is an important task in computer vision aimed at improving the semantic understanding of the visual world. In this task, the model needs to detect objects and predict visual relationships between them. Most of the existing models predict relationships in parallel assuming their independence. While there are different ways to capture these dependencies, we explore a conditional approach motivated by the sequence-to-sequence (Seq2Seq) formalism. Different from the previous research, our proposed model predicts visual relationships one at a time in an autoregressive manner by explicitly conditioning on the already predicted relationships. Drawing from translation models in NLP, we propose an encoder-decoder model built using Transformers where the encoder captures global context and long range interactions. The decoder then makes sequential predictions by conditioning on the scene graph constructed so far. In addition, we introduce a novel reinforcement learning-based training strategy tailored to Seq2Seq scene graph generation. By using a self-critical policy gradient training approach with Monte Carlo search we directly optimize for the (mean) recall metrics and bridge the gap between training and evaluation. Experimental results on two public benchmark datasets demonstrate that our Seq2Seq learning approach achieves strong empirical performance, outperforming previous state-of-the-art, while remaining efficient in terms of training and inference time. Full code for this work is available here: https://github.com/layer6ai-labs/SGG-Seq2Seq.