Reviewerly/exHarmony huggingface dataset:
Egocentric video summarisation via purpose-oriented frame scoring and selection

Existing video summarisation techniques are quite generic in nature, since they generally overlook the important aspect of what actual purpose the summary will be serving. In sharp contrast with this mainstream work, it can be acknowledged that there are many possible purposes the same videos can be summarised for. Accordingly, we consider a novel perspective: summaries with a purpose. This work is an attempt to both, call the attention on this neglected aspect of video summarisation research, and to illustrate it and explore it with two concrete purposes, focusing on first-person-view videos. The proposed purpose-oriented summarisation techniques are framed under the common (frame-level) scoring and selection paradigm, and have been tested on two egocentric datasets, BEOID and EGTEA-Gaze+. The necessary purpose-specific evaluation metrics are also introduced. The proposed approach is compared with two purpose-agnostic summarisation baselines. On the one hand, a partially agnostic method uses the scores obtained by the proposed approach, but follows a standard generic frame selection technique. On the other hand, the fully agnostic method do not use any purpose-based information, and relies on generic concepts such as diversity and representativeness. The results of the experimental work show that the proposed approaches compare favourably with respect to both baselines. More specifically, the purpose-specific approach generally produces summaries with the best compromise between summary lengths and favourable purpose-specific metrics. Interestingly, it is also observed that results of the partially-agnostic baseline tend to be better than those of the fully-agnostic one. These observations provide strong evidence on the advantage and relevance of purpose-specific summarisation techniques and evaluation metrics, and encourage further work on this important subject.
Blockchain for V2X: A Taxonomy of Design Use Cases and System Requirements

Vehicles today contain a multitude of sensors creating vast amounts of data. For many applications, these data need to be shared with other entities so that they can also utilize it. Vehicle-to-everything (V2X) is the amalgamation of all potential vehicle communication systems. V2X technologies are enabling many smart-vehicle applications, such as autonomous vehicles. However, in utilizing these data from external entities, vehicles rely on the availability and trustworthiness of centralized entities who may be able to delete, forge, leak, or otherwise tamper with the underlying data. Blockchain technology provides a decentralized mechanism to allow vehicles to validate data they receive in a trustless manner. This paper explores potential applications of blockchain technology in the V2X space, categorizing and analyzing use cases based on their underlying blockchain requirements. It then uses this analysis to determine the key requirements behind an effective V2X blockchain.
Smart(Sampling)Augment: Optimal and Efficient Data Augmentation for Semantic Segmentation

Data augmentation methods enrich datasets with augmented data to improve the performance of neural networks. Recently, automated data augmentation methods have emerged, which automatically design augmentation strategies. The existing work focuses on image classification and object detection, whereas we provide the first study on semantic image segmentation and introduce two new approaches: SmartAugment and SmartSamplingAugment. SmartAugment uses Bayesian Optimization to search a rich space of augmentation strategies and achieves new state-of-the-art performance in all semantic segmentation tasks we consider. SmartSamplingAugment, a simple parameter-free approach with a fixed augmentation strategy, competes in performance with the existing resource-intensive approaches and outperforms cheap state-of-the-art data augmentation methods. Furthermore, we analyze the impact, interaction, and importance of data augmentation hyperparameters and perform ablation studies, which confirm our design choices behind SmartAugment and SmartSamplingAugment. Lastly, we will provide our source code for reproducibility and to facilitate further research.
Double layered Fridrich structure to conserve medical data privacy using quantum cryptosystem

To handle pandemic situations like Corona, medical practitioners and researchers collaboratively will work together and transfer medical reports across the globe. Subsequently, it is important to preserve those reports from unauthorised access by providing utmost confidentiality, integrity and authentication. Quantum cryptography depends on quantum mechanics principles such as Heisenberg uncertainty and Photon polarisation, which aims in preventing the bootlegging of the encoded secret data in the quantum state. Quantum Algorithms are recently gaining wide attention among researcher's worldwide in optimising the protocols with adaptive learning and quadratic speed up compared to classical ones. Hence, a quantum assisted medical image encryption algorithm has been devised in protecting the shared reports in this paper. Initially, the medical images in Digital Imaging in Communication and Medicine (DICOM) format will be converted into Novel Enhanced Quantum Image Representation (NEQR) to survive in the quantum environment. Further, combined chaotic generators were utilised to generate random keys, ensuring a larger chaotic range and stability to withstand open communal channel attacks. Furthermore, double-layered Fridrich operations (Confusion and Diffusion) were carried out using the generated keys by adopting Quantum reversible gates. Finally, intra and inter bit permutations were conceded to provide a complete cryptic cipher image with uniform distribution of bits to resist the vulnerable attacks and enhance parallel data encryption procedure speed. Statistical and Differential attack analyses were estimated and compared with state of the art to justify the robustness of the proposed quantum cryptosystem. Also, complexity analysis has been derived to prove the proposed quantum algorithm's computational speedup compared to the classical part.
Parallel Implementation of PIPO Block Cipher on 32-bit RISC-V Processor

In ICISC 2020, 64-bit PIPO lightweight block cipher was presented. The main design goals of PIPO are the implementation-friendly and the ease of designing side-channel protection techniques. Until now, the performance of PIPO has been investigated only on 8-bit AVR environment. Thus, optimization strategies on various embedded devices need to be investigated for PIPO’s wide usage in various IoT applications. For filling this gap, in this paper, we present an optimized software implementation of PIPO cipher on 32-bit RISC-V processor being widely considered as an alternative embedded processor for ARM processors. For optimal performance, we propose several novel techniques: optimal register scheduling for minimizing the number of memory accesses, data parallel processing approach by using 32-bit register set, and combined internal process. In result, our software shows 128.5% improved performance on PIPO-64/128 basis than the simple ported version of RISC-V. In addition, our implementation showed 393.52% improvement over the encryption performance of the PIPO reference code, despite including the key scheduling process. As far as we know, this is the first optimal implementation of PIPO block cipher on RISC-V environment.
Do social identity and cognitive diversity correlate in environmental stakeholders? A novel approach to measuring cognitive distance within and between groups

Groups with higher cognitive diversity, i.e. variations in how people think and solve problems, are thought to contribute to improved performance in complex problem-solving. However, embracing or even engineering adequate cognitive diversity is not straightforward and may even jeopardize social inclusion. In response, those that want to promote cognitive diversity might make a simplified assumption that there exists a link between identity diversity, i.e. range of social characteristics, and variations in how people perceive and solve problems. If this assumption holds true, incorporating diverse identities may concurrently achieve cognitive diversity to the extent essential for complex problem-solving, while social inclusion is explicitly acknowledged. However, currently there is a lack of empirical evidence to support this hypothesis in the context of complex social-ecological systems-a system wherein human and environmental dimensions are interdependent, where common-pool resources are used or managed by multiple types of stakeholders. Using a fisheries example, we examine the relationship between resource stakeholders' identities and their cognitive diversity. We used cognitive mapping techniques in conjunction with network analysis to measure cognitive distances within and between stakeholders of various social types (i.e., identities). Our results empirically show that groups with higher identity diversity also demonstrate more cognitive diversity, evidenced by disparate characteristics of their cognitive maps that represent their understanding of fishery dynamics. These findings have important implications for sustainable management of common-pool resources, where the inclusion of diverse stakeholders is routine, while our study shows it may also achieve higher cognitive coverage that can potentially lead to more complete, accurate, and innovative understanding of complex resource dynamics.
Evaluating Grasping Visualizations and Control Modes in a VR Game

A primary goal of the Virtual Reality ( VR ) community is to build fully immersive and presence-inducing environments with seamless and natural interactions. To reach this goal, researchers are investigating how to best directly use our hands to interact with a virtual environment using hand tracking. Most studies in this field require participants to perform repetitive tasks. In this article, we investigate if results of such studies translate into a real application and game-like experience. We designed a virtual escape room in which participants interact with various objects to gather clues and complete puzzles. In a between-subjects study, we examine the effects of two input modalities (controllers vs. hand tracking) and two grasping visualizations (continuously tracked hands vs. virtual hands that disappear when grasping) on ownership, realism, efficiency, enjoyment, and presence. Our results show that ownership, realism, enjoyment, and presence increased when using hand tracking compared to controllers. Visualizing the tracked hands during grasps leads to higher ratings in one of our ownership questions and one of our enjoyment questions compared to having the virtual hands disappear during grasps as is common in many applications. We also confirm some of the main results of two studies that have a repetitive design in a more realistic gaming scenario that might be closer to a typical user experience.
Fast Implementation of SHA-3 in GPU Environment

Recently, Graphic Processing Units (GPUs) have been widely used for general purpose applications such as machine learning applications, acceleration of cryptographic applications (especially, blockchains), etc.The development of CUDA makes this General-Purpose computing on GPU possible.In particular, currently GPU technology has been widely used for server-side applications so as to provide fast and efficient service to a number of clients.In other words, servers need to process a large amount of user data and execute authentication process.Verifying the integrity of transmitted data is essential for ensuring that the data is not modified during transmission.Hash functions are the cryptographic algorithm which can verify the integrity of data and there are SHA-1, SHA-2, and SHA-3 standard hash functions.In 2015, Keccak algorithm was selected for SHA-3 competition by NIST.However, until now, software implementations of SHA-3 have not provided enough performance for various applications.In addition, SHA-3 and SHAKE using SHA-3 are being used in many Post-Quantum Cryptosystems (PQC) submitted to NIST PQC competition.Therefore, SHA-3 optimization research is required in the software environment.We propose an optimized SHA-3 software implementation on GPU environment.For performance efficiency, we propose several techniques including optimization of SHA-3 internal process, inline PTX optimization, optimized memory usage, and the application of asynchronous CUDA stream.As a result of applying the proposed optimization method, our SHA-3(512) (resp.SHA-3(256)) implementation without CUDA stream provides a maximum throughput of 88.51 Gb/s (resp.171.62 Gb/s) on RTX2080Ti GPU.Furthermore, without the application of CUDA stream, our SHA-3(512) software on GTX1070 provides about 49.73% improved throughput compared with the previous best work on GTX1080, which shows the superiority of our proposed optimization methods.Our optimized SHA-3 software on GPU can be efficiently used for block-chain applications and several PQCs (especially, key generation process in Lattice-based cryptosystems).
SnipeDet: Attention-guided pyramidal prediction kernels for generic object detection

Using single-scale prediction kernels or Region of Interest (RoI) pooling in the prediction modules of modern object detectors is not very successful in matching different scales of objects. State-of-the-art detectors with the feature pyramid structure built on different resolutions of feature maps can help alleviate this problem. Although with this structure, single-scale prediction kernels or RoI pooling still struggles to detect small objects, and simultaneously, the former continues to encounter the misalignment problem on very large objects. In this paper, we propose the attention-guided pyramidal prediction kernels module with a customized IoU-adaptive loss function to deal with the misalignment problem between the prediction module and different scales of objects. To mitigate the effect of heavy detection head, we also introduce the salient object regions recognition module to identify these regions that have strong object cues. Additionally, interleaved subsampling, as the proposed feature enhancement approach, is used to generate highly discriminative feature representations. We refer to the detection framework constituted by these proposed methods as SnipeDet. Results show that SnipeDet achieves 41.1 AP at the speed of 15.4 FPS on the MS COCO test-dev set with 512 × 512 input images, which outperforms state-of-the-art one-stage detectors and has a better trade-off between speed and accuracy.
Adversarial Robustness of Deep Learning: Theory, Algorithms, and Applications

This tutorial aims to introduce the fundamentals of adversarial robustness of deep learning, presenting a well-structured review of up-to-date techniques to assess the vulnerability of various types of deep learning models to adversarial examples. This tutorial will particularly highlight state-of-the-art techniques in adversarial attacks and robustness verification of deep neural networks (DNNs). We will also introduce some effective countermeasures to improve robustness of deep learning models, with a particular focus on adversarial training. We aim to provide a comprehensive overall picture about this emerging direction and enable the community to be aware of the urgency and importance of designing robust deep learning models in safety-critical data analytical applications, ultimately enabling the end-users to trust deep learning classifiers. We will also summarize potential research directions concerning the adversarial robustness of deep learning, and its potential benefits to enable accountable and trustworthy deep learning-based data analytical systems and applications.
Cross-attention-map-based regularization for adversarial domain adaptation

In unsupervised domain adaptation (UDA), many efforts are taken to pull the source domain and the target domain closer by adversarial training. Most methods focus on aligning distributions or features between the source domain and the target domain. However, little attention is paid to the interaction between finer-grained levels, such as classes or samples of the two domains. In contrast to UDA, another transfer learning task, i.e., few-shot learning (FSL), takes full advantage of the finer-grained-level alignment. Many FSL methods implement the interaction between samples of support sets and query sets, leading to significant improvements. We wonder whether we can get some inspiration from these methods and bring such ideas of FSL to UDA. To this end, we first take a closer look at the differences between FSL and UDA and bridge the gap between them by high-confidence sample selection (HCSS). Then we propose cross-attention map generation module (CAMGM) to interact samples selected by HCSS. Moreover, we propose a simple but efficient method called cross-attention-map-based regularization (CAMR) to regularize the feature maps generated by the feature extractor. Experiments on three challenging datasets demonstrate that CAMR can bring solid improvements when added to the original objective. More specifically, the proposed CAMR can outperform original methods by 1% to 2% in most tasks without bells and whistles.
Toward Responsible AI: An Overview of Federated Learning for User-centered Privacy-preserving Computing

With the rapid advances of Artificial Intelligence (AI) technologies and applications, an increasing concern is on the development and application of responsible AI technologies. Building AI technologies or machine-learning models often requires massive amounts of data, which may include sensitive, user private information to be collected from different sites or countries. Privacy, security, and data governance constraints rule out a brute force process in the acquisition and integration of these data. It is thus a serious challenge to protect user privacy while achieving high-performance models. This article reviews recent progress of federated learning in addressing this challenge in the context of privacy-preserving computing. Federated learning allows global AI models to be trained and used among multiple decentralized data sources with high security and privacy guarantees, as well as sound incentive mechanisms. This article presents the background, motivations, definitions, architectures, and applications of federated learning as a new paradigm for building privacy-preserving, responsible AI ecosystems.
Measuring the Perceived Three-Dimensional Location of Virtual Objects in Optical See-Through Augmented Reality

For optical see-through augmented reality (AR), a new method for measuring the perceived three-dimensional location of virtual objects is presented, where participants verbally report a virtual object’s location relative to both a vertical and horizontal grid. The method is tested with a small (1.95 × 1.95 × 1.95 cm) virtual object at distances of 50 to 80 cm, viewed through a Microsoft HoloLens 1 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">st</sup> generation AR display. Two experiments examine two different virtual object designs, whether turning in a circle between reported object locations disrupts HoloLens tracking, and whether accuracy errors, including a rightward bias and underestimated depth, might be due to systematic errors that are restricted to a particular display. Turning in a circle did not disrupt HoloLens tracking, and testing with a second display did not suggest systematic errors restricted to a particular display. Instead, the experiments are consistent with the hypothesis that, when looking downwards at a horizontal plane, HoloLens 1 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">st</sup> generation displays exhibit a systematic rightward perceptual bias. Precision analysis suggests that the method could measure the perceived location of a virtual object within an accuracy of less than 1 mm.
From the whole to detail: Progressively sampling discriminative parts for fine-grained recognition

Fine-grained image recognition puts forward a special challenge due to the difficulties of distinguishing subtle inter-class differences and large intra-class variances. Existing weakly supervised approaches tend to capture the most discriminative regions, thereby guiding network to learn fine-grained features. However, current methods neglect the correlation between object and details, where object localization is conductive to part detection. In addition, they generally not only need heavy computational cost to find details with auxiliary subnet or selective strategy, but also require well-designed bounding boxes which are inflexible for different scale targets. In this paper, we propose a more lightweight framework to progressively sampling discriminative parts for learning details from coarse-scale to fine-scale, without any pre-designed bounding boxes. Our method first amplifies the object (e.g., bird, car) from the original image in the light of class visual patterns, then a self-adaptive region sampler applied to detect most informative regions from attention maps to learn fine-grained representations. The framework consists of three streams, i.e., the whole, the object and the detail respectively, thus hierarchical features can be preserved and learned. Furthermore, our approach can be trained end-to-end in a weakly supervised manner, and few computational costs are needed at inference phase. Comprehensive experiments and ablation studies demonstrate that the proposed method obtains competitive performance on three benchmarks.
Multi-Scale Fusion With Matching Attention Model: A Novel Decoding Network Cooperated With NAS for Real-Time Semantic Segmentation

This paper proposes a real-time multi-scale semantic segmentation network (MsNet). MsNet is a combination of our novel multi-scale fusion with matching attention model (MFMA) as the decoding network and the network searched by asymptotic neural architecture search (ANAS) or MobileNetV3 as the encoding network. The MFMA not only extracts low level spatial features from multi-scale inputs but also decodes the contextual features extracted by ANAS. Specifically, considering the advantages and disadvantages of the addition fusion and concatenation fusion, we design multi-scale fusion (MF) that balances speed and accuracy. Then we creatively design two matching attention mechanisms (MA), including matching attention with low calculation (MALC) mechanism and matching attention with strong global context modeling (MASG) mechanism, to match varying resolutions and information of features at different levels of a network. Besides, the ANAS performs the deep neural network search by employing an asymptotic method and provide an efficient encoding network for MsNet, releasing researchers from those tedious mechanical trials. Through extensive experiments, we prove that MFMA, which can be applied to numerous recognition tasks, possesses excellent decoding ability. And we demonstrate the effectiveness and necessity of implementing the “matching” attention mechanism. Finally, the proposed two versions, MsNet_ANAS and MsNet_M achieve a new state-of-the-art trade-off between accuracy and speed on the CamVid and Cityscapes datasets. More remarkably, on the Nvidia Tesla V100 GPU, our MsNet_ANAS achieves 74.1% mIoU with the speed of 184.2 FPS on the CamVid while 72.9% mIoU with the speed of 119.9 FPS on the Cityscapes.