Reviewerly/exHarmony huggingface dataset:
Crowd anomaly detection using Aggregation of Ensembles of fine-tuned ConvNets

Anomaly detection in crowded scenes plays a crucial role in automatic video surveillance to avert any casualty in the areas witnessing the high amount of footfalls. The key challenge for automatically classifying the anomalies in crowd image is the usage of feature set and techniques which can be replicated in every crowded scenario. In this paper, we propose a novel concept of Aggregation of Ensembles (AOE) for detecting an anomaly in video data showing crowded scenes, which leverage the existing capability of pre-trained ConvNets and a pool of classifiers. The proposed approach uses an ensemble of different fine-tuned Convolutional Neural Networks (CNN) based on the hypothesis that different CNN architectures learn different levels of semantic representation from crowd videos and thus an ensemble of CNNs will enable enriched feature sets to be extracted. The proposed AOE concept utilizes the fine-tuned ConvNets as fixed feature extractors to train variants of SVM classifier and then the posterior probabilities are fused to predict the anomaly in the crowd frame sequences. The experimental results show that the proposed Aggregation of Ensembles fine-tuned CNNs of various architectures achieve a higher accuracy in comparison with other established methods on benchmark datasets.
Decentralized Resource Allocation for Video Transcoding and Delivery in Blockchain-Based System With Mobile Edge Computing

The blockchain-based video systems are designed to build a decentralized and flexible video ecosystem by enabling a direct interaction among users, video providers (VPs), and service providers. In blockchain-based video systems, the heterogeneous qualities and formats of the video streams usually require massive computational resources to transcode them into different versions and formats to meet distinct requirements of users. However, current blockchains cannot handle massive and heterogeneous video streaming due to limited computing capacity and long transaction times. To deal with this issue, in this paper, leveraging mobile edge computing (MEC) technology, we propose a blockchain-based MEC architecture, where small base stations (SBSs) allocate their computation as well as communication resources for providing video streaming in a distributed and secure manner. Moreover, to improve the operation efficiency, we use a series of smart contracts to enable a self-organized video transcoding and delivery service without a centralized controller. Then, users, SBSs, and VP could adjust their strategies according to the transactional information on blockchain. Moreover, we formulate the video transcoding and delivery problem as a three-stage Stackelberg game. We analyze the sub-game equilibrium in each stage and the interplays of the three-stage game. Last, we propose an iterative algorithm to solve the problem. Simulation results show that the proposed approach could obtain the good performance in terms of average time to finality (TTF), average access delay, and network cost.
FANTrack: 3D Multi-Object Tracking with Feature Association Network

We propose a data-driven approach to online multi-object tracking (MOT) that uses a convolutional neural network (CNN) for data association in a tracking-by-detection framework. The problem of multi-target tracking aims to assign noisy detections to a-priori unknown and time-varying number of tracked objects across a sequence of frames. A majority of the existing solutions focus on either tediously designing cost functions or formulating the task of data association as a complex optimization problem that can be solved effectively. Instead, we exploit the power of deep learning to formulate the data association problem as inference in a CNN. To this end, we propose to learn a similarity function that combines cues from both image and spatial features of objects. Our solution learns to perform global assignments in 3D purely from data, handles noisy detections and varying number of targets, and is easy to train. We evaluate our approach on the challenging KITTI dataset and show competitive results. Our code is available at https://git.uwaterloo.ca/wise-lab/fantrack.
Evaluation and Demonstration of Blockchain Applicability Framework

Blockchain technology has been gaining great interest from a variety of industry sectors, including financial, food processing, and power and energy markets. Realizing the strength of blockchain technology beyond the successful application in the cryptocurrency arena, researchers have been evaluating and using blockchain for applications such as supply chain management, transactive industry (both financial and energy), system integrity, device cybersecurity, identity management, and much more. One of the unique elements of the blockchain technology that made it such a captivating technology to researchers is its plethora of features. Some of the features include smart contracts, cryptocurrency and tokenizing, immutable distributed ledger, cryptographic hashing, and digital signature. In addition, there are multiple types of blockchains, such as permissioned/private and permissionless/public, and various consensus models, such as proof-of-work, proof-of-authority, proof-of-burn, and proof-of-stake. Therefore, it is often nontrivial to determine if an application requires a blockchain. If so, what kind of blockchain and consensus is most appropriate? This article discusses the blockchain applicability framework (BAF), which was specifically designed with the purpose to answer those questions. The BAF is divided into five domains, 18 subdomains, and about 100 controls. It is designed to ingest detailed user requirements to perform a weighted evaluation that is built on mathematical constructs to determine the ideal combination of blockchain that is appropriate for an application. Along with the core logical formulation of the BAF, this article depicts the efficacy of the BAF through two use cases.
A Lightweight Hash-Based Blockchain Architecture for Industrial IoT

Blockchain is a technology that can ensure data integrity in a distributed network, and it is actively applied in various fields. Recently, blockchain is gaining attention due to combining with the Internet of Things (IoT) technology in the industrial field. Moreover, many researchers have proposed the Industrial IoT (IIoT) architecture with blockchain for data integrity and efficient management. The IIoT network consists of many heterogeneous devices (e.g., sensors, actuators, and programmable logic controllers (PLC)) with resources-constrained, and the availability of the network must be preferentially considered. Therefore, applying the existed blockchain technology is still challenging. There are some results about the technique of constructing blockchain lightly to solve this challenge. However, in these results, the analysis in perspective of cryptographic performance (area, throughput, and power consumption) has not been considered sufficiently, or only focused on the architecture of the blockchain network. The blockchain technology is based on cryptographic techniques, and the main part is a cryptographic hash function. Therefore, if we construct the blockchain-based IIoT architecture, we have to consider the performance of the hash function. Many lightweight hash functions have been proposed recently for the resource-constrained environment, and it can also be used to the blockchain. Therefore, in this paper, we analyze the considerations of lightweight blockchain for IIoT. Also, we conduct an analysis of lightweight hash for blockchain, and propose a new lightweight hash-based blockchain architecture that can change the hash algorithm used for mining adjust to network traffic.
Attention GANs: Unsupervised Deep Feature Learning for Aerial Scene Classification

With the development of deep learning, supervised feature learning methods have achieved prominent performance in the field of aerial scene classification. However, supervised feature learning methods require a large amount of labeled training data. To address this limitation, in this article, a novel unsupervised deep feature learning method, namely, Attention generative adversarial networks (Attention GANs), is proposed for aerial scene classification. First, Attention GANs integrates the attention mechanism into GANs to enhance the representation power of the discriminator. Then, to obtain contextual information, a context-aggregation-based feature fusion architecture is designed in the discriminator. Furthermore, the generator and discriminator losses are improved on basis of the Relativistic GAN. At the same time, a content loss is formed by using the feature representations from the context-aggregation-based feature fusion architecture. In the experiments, our Attention GANs is evaluated via comprehensive experiments with four publicly available remote sensing scene data sets, i.e., the UC-Merced data set with 21 scene classes, the RSSCN7 data set with 7 scene classes, the AID data set with 30 scene classes, and the NWPU-RESISC45 data set with 45 scene classes. Experimental results demonstrate that our Attention GANs can obtain the best performance compared with the state-of-the-art methods.
Interact as You Intend: Intention-Driven Human-Object Interaction Detection

The recent advances in instance-level detection tasks lay strong foundation for genuine comprehension of the visual scenes. However, the ability to fully comprehend a social scene is still in its preliminary stage. In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications. To undertake social tasks interacting with objects, humans direct their attention and move their body based on their intention. Based on this observation, we provide a unique computational perspective to explore human intention in HOI detection. Specifically, the proposed human intention-driven HOI detection (iHOI) framework models human pose with the relative distances from body joints to the object instances. It also utilizes human gaze to guide the attended contextual regions in a weakly-supervised setting. In addition, we propose a hard negative sampling strategy to address the problem of mis-grouping. We perform extensive experiments on two benchmark datasets, namely V-COCO and HICO-DET. The efficacy of each proposed component has also been validated.
DeepAtlas: Joint Semi-supervised Learning of Image Registration and Segmentation

Deep convolutional neural networks (CNNs) are state-of-the-art for semantic image segmentation, but typically require many labeled training samples. Obtaining 3D segmentations of medical images for supervised training is difficult and labor intensive. Motivated by classical approaches for joint segmentation and registration we therefore propose a deep learning framework that jointly learns networks for image registration and image segmentation. In contrast to previous work on deep unsupervised image registration, which showed the benefit of weak supervision via image segmentations, our approach can use existing segmentations when available and computes them via the segmentation network otherwise, thereby providing the same registration benefit. Conversely, segmentation network training benefits from the registration, which essentially provides a realistic form of data augmentation. Experiments on knee and brain 3D magnetic resonance (MR) images show that our approach achieves large simultaneous improvements of segmentation and registration accuracy (over independently trained networks) and allows training high-quality models with very limited training data. Specifically, in a one-shot-scenario (with only one manually labeled image) our approach increases Dice scores (%) over an unsupervised registration network by 2.7 and 1.8 on the knee and brain images respectively.
High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning

Edge machine learning involves the deployment of learning algorithms at the wireless network edge so as to leverage massive mobile data for enabling intelligent applications. The mainstream edge learning approach, federated learning, has been developed based on distributed gradient descent. Based on the approach, stochastic gradients are computed at edge devices and then transmitted to an edge server for updating a global AI model. Since each stochastic gradient is typically high-dimensional, communication overhead becomes a bottleneck for edge learning. To address this issue, we propose a novel framework of hierarchical gradient quantization and study its effect on the learning performance. First, the framework features a practical hierarchical architecture for decomposing the stochastic gradient into its norm and normalized block gradients, and efficiently quantizes them using a uniform quantizer and a low-dimensional Grassmannian codebook, respectively. Subsequently, the quantized normalized block gradients are scaled and cascaded to yield the quantized normalized stochastic gradient using a socalled hinge vector, which is compressed using another low-dimensional Grassmannian quantizer designed under the criterion of minimum distortion. The other feature of the framework is a bit-allocation scheme for reducing the distortion, which divides the total quantization bits to determine the resolutions of low-dimensional quantizers. The framework is proved to guarantee model convergency by analyzing the convergence rate as a function of quantization bits. Furthermore, by simulation, our design is shown to substantially reduce the communication overhead compared with the state-of-the-art signSGD scheme, while achieving similar learning accuracies.
W2VV++

Ad-hoc video search (AVS) is an important yet challenging problem in multimedia retrieval. Different from previous concept-based methods, we propose a fully deep learning method for query representation learning. The proposed method requires no explicit concept modeling, matching and selection. The backbone of our method is the proposed W2VV++ model, a super version of Word2VisualVec (W2VV) previously developed for visual-to-text matching. W2VV++ is obtained by tweaking W2VV with a better sentence encoding strategy and an improved triplet ranking loss. With these simple yet important changes, W2VV++ brings in a substantial improvement. As our participation in the TRECVID 2018 AVS task and retrospective experiments on the TRECVID 2016 and 2017 data show, our best single model, with an overall inferred average precision (infAP) of 0.157, outperforms the state-of-the-art. The performance can be further boosted by model ensemble using late average fusion, reaching a higher infAP of 0.163. With W2VV++, we establish a new baseline for ad-hoc video search.
Image Synthesis From Reconfigurable Layout and Style

Despite remarkable recent progress on both unconditional and conditional image synthesis, it remains a long- standing problem to learn generative models that are capable of synthesizing realistic and sharp images from re- configurable spatial layout (i.e., bounding boxes + class labels in an image lattice) and style (i.e., structural and appearance variations encoded by latent vectors), especially at high resolution. By reconfigurable, it means that a model can preserve the intrinsic one-to-many mapping from a given layout to multiple plausible images with different styles, and is adaptive with respect to perturbations of a layout and style latent code. In this paper, we present a layout- and style-based architecture for generative adversarial networks (termed LostGANs) that can be trained end-to-end to generate images from reconfigurable layout and style. Inspired by the vanilla StyleGAN, the proposed LostGAN consists of two new components: (i) learning fine-grained mask maps in a weakly-supervised manner to bridge the gap between layouts and images, and (ii) learning object instance-specific layout-aware feature normalization (ISLA-Norm) in the generator to realize multi-object style generation. In experiments, the proposed method is tested on the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained. The code and pretrained models are available at https://github.com/iVMCL/LostGANs.
SBSGAN: Suppression of Inter-Domain Background Shift for Person Re-Identification

Cross-domain person re-identification (re-ID) is challenging due to the bias between training and testing domains. We observe that if backgrounds in the training and testing datasets are very different, it dramatically introduces difficulties to extract robust pedestrian features, and thus compromises the cross-domain person re-ID performance. In this paper, we formulate such problems as a background shift problem. A Suppression of Background Shift Generative Adversarial Network (SBSGAN) is proposed to generate images with suppressed backgrounds. Unlike simply removing backgrounds using binary masks, SBSGAN allows the generator to decide whether pixels should be preserved or suppressed to reduce segmentation errors caused by noisy foreground masks. Additionally, we take ID-related cues, such as vehicles and companions into consideration. With high-quality generated images, a Densely Associated 2-Stream (DA-2S) network is introduced with Inter Stream Densely Connection (ISDC) modules to strengthen the complementarity of the generated data and ID-related cues. The experiments show that the proposed method achieves competitive performance on three re-ID datasets, i.e., Market-1501, DukeMTMC-reID, and CUHK03, under the cross-domain person re-ID scenario.
Resource Scheduling in Cloud Computing Based on a Hybridized Whale Optimization Algorithm

The cloud computing paradigm, as a novel computing resources delivery platform, has significantly impacted society with the concept of on-demand resource utilization through virtualization technology. Virtualization enables the usage of available physical resources in a way that multiple end-users can share the same underlying hardware infrastructure. In cloud computing, due to the expectations of clients, as well as on the providers side, many challenges exist. One of the most important nondeterministic polynomial time (NP) hard challenges in cloud computing is resource scheduling, due to its critical impact on the cloud system performance. Previously conducted research from this domain has shown that metaheuristics can substantially improve cloud system performance if they are used as scheduling algorithms. This paper introduces a hybridized whale optimization algorithm, that falls into the category of swarm intelligence metaheuristics, adapted for tackling the resource scheduling problem in cloud environments. To more precisely evaluate performance of the proposed approach, original whale optimization was also adapted for resource scheduling. Considering the two most important mechanisms of any swarm intelligence algorithm (exploitation and exploration), where the efficiency of a swarm algorithm depends heavily on their adjusted balance, the original whale optimization algorithm was enhanced by addressing its weaknesses of inappropriate exploitation–exploration trade-off adjustments and the premature convergence. The proposed hybrid algorithm was first tested on a standard set of bound-constrained benchmarks with the goal to more accurately evaluate its performance. After, simulations were performed using two different resource scheduling models in cloud computing with real, as well as with artificial data sets. Simulations were performed on the robust CloudSim platform. A hybrid whale optimization algorithm was compared with other state-of-the-art metaheurisitcs and heuristics, as well as with the original whale optimization for all conducted experiments. Achieved results in all simulations indicate that the proposed hybrid whale optimization algorithm, on average, outperforms the original version, as well as other heuristics and metaheuristics. By using the proposed algorithm, improvements in tackling the resource scheduling issue in cloud computing have been established, as well enhancements to the original whale optimization implementation.
Immersive Versus Non-immersive Experience: Exploring the Feasibility of Memory Assessment Through 360° Technology

Episodic memory is essential to effectively perform a number of daily activities, as it enables individuals to consciously recall experiences within their spatial and temporal environments. Virtual Reality (VR) serves as an efficacious instrument to assess cognitive functions like attention and memory. Previous studies have adopted computer-simulated VR to assess memory, which realized greater benefits compared to traditional procedures (paper and pencil). One of the most recent trends of immersive VR experiences is the 360° technology. In order to evaluate its capabilities, this study aims to compare memory performance through two tasks: immersive task and non-immersive task. These tasks differ based on the participant's view of the 360° picture: (1) head-mounted display (HMD) for immersive task and (2) tablet for non-immersive task. This study seeks to compare how memory is facilitated in both the 360° immersive picture as well as the non-immersive 360° picture. A repeated measure design was carried out in a sample of 42 participants, randomized into two groups of 21. Group 1 first observed Picture A (immersive) followed by Picture B (non-immersive) while Group 2 began with Picture B and then looked at Picture A. Each 360° picture contains specific items with some items appearing in both. Memory evaluation is assessed immediately after the exposure (recall task), then again after a 10-min delay (recognition task). Results reveal that Group 1, which began with the immersive task, demonstrated stronger memory performance in the long term as compared to Group 2, which began with the non-immersive task. Preliminary data ultimately supports the efficacy of the 360° technology in evaluating cognitive function.
Few-Shot Learning With Global Class Representations

In this paper, we propose to tackle the challenging few-shot learning (FSL) problem by learning global class representations using both base and novel class training samples. In each training episode, an episodic class mean computed from a support set is registered with the global representation via a registration module. This produces a registered global class representation for computing the classification loss using a query set. Though following a similar episodic training pipeline as existing meta learning based approaches, our method differs significantly in that novel class training samples are involved in the training from the beginning. To compensate for the lack of novel class training samples, an effective sample synthesis strategy is developed to avoid overfitting. Importantly, by joint base-novel class training, our approach can be easily extended to a more practical yet challenging FSL setting, i.e., generalized FSL, where the label space of test data is extended to both base and novel classes. Extensive experiments show that our approach is effective for both of the two FSL settings.