Reviewerly/exHarmony huggingface dataset:
FVC: A New Framework towards Deep Video Compression in Feature Space

Learning based video compression attracts increasing attention in the past few years. The previous hybrid coding approaches rely on pixel space operations to reduce spatial and temporal redundancy, which may suffer from inaccurate motion estimation or less effective motion compensation. In this work, we propose a feature-space video coding network (FVC) by performing all major operations (i.e., motion estimation, motion compression, motion compensation and residual compression) in the feature space. Specifically, in the proposed deformable compensation module, we first apply motion estimation in the feature space to produce motion information (i.e., the offset maps), which will be compressed by using the auto-encoder style network. Then we perform motion compensation by using deformable convolution and generate the predicted feature. After that, we compress the residual feature between the feature from the current frame and the predicted feature from our deformable compensation module. For better frame reconstruction, the reference features from multiple previous reconstructed frames are also fused by using the nonlocal attention mechanism in the multi-frame feature fusion module. Comprehensive experimental results demonstrate that the proposed framework achieves the state-of-the-art performance on four benchmark datasets including HEVC, UVG, VTL and MCL-JCV.
Joint Noise-Tolerant Learning and Meta Camera Shift Adaptation for Unsupervised Person Re-Identification

This paper considers the problem of unsupervised person re-identification (re-ID), which aims to learn discriminative models with unlabeled data.One popular method is to obtain pseudo-label by clustering and use them to optimize the model.Although this kind of approach has shown promising accuracy, it is hampered by 1) noisy labels produced by clustering and 2) feature variations caused by camera shift. The former will lead to incorrect optimization and thus hinders the model accuracy. The latter will result in assigning the intra-class samples of different cameras to different pseudo-label, making the model sensitive to camera variations. In this paper, we propose a unified framework to solve both problems. Concretely, we propose a Dynamic and Symmetric Cross-Entropy loss (DSCE) to deal with noisy samples and a camera-aware meta-learning algorithm (MetaCam) to adapt camera shift. DSCE can alleviate the negative effects of noisy samples and accommodate the change of clusters after each clustering step. MetaCam simulates cross-camera constraint by splitting the training data into meta-train and meta-test based on camera IDs.With the interacted gradient from meta-train and meta-test, the model is enforced to learn camera-invariant features.Extensive experiments on three re-ID benchmarks show the effectiveness and the complementary of the proposed DSCE and MetaCam.Our method outperforms the state-of-the-art methods on both fully unsupervised re-ID and unsupervised domain adaptive re-ID.
Unsupervised Scale-Consistent Depth Learning from Video

We propose a monocular depth estimation method SC-Depth, which requires only unlabelled videos for training and enables the scale-consistent prediction at inference time. Our contributions include: (i) we propose a geometry consistency loss, which penalizes the inconsistency of predicted depths between adjacent views; (ii) we propose a self-discovered mask to automatically localize moving objects that violate the underlying static scene assumption and cause noisy signals during training; (iii) we demonstrate the efficacy of each component with a detailed ablation study and show high-quality depth estimation results in both KITTI and NYUv2 datasets. Moreover, thanks to the capability of scale-consistent prediction, we show that our monocular-trained deep networks are readily integrated into ORB-SLAM2 system for more robust and accurate tracking. The proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training. Finally, we provide several demos for qualitative evaluation. The source code is released on GitHub.
Exploring smart construction objects as blockchain oracles in construction supply chain management

Blockchain technology has attracted the interest of the global construction industry for its potential to enhance the transparency, traceability, and immutability of construction data and enables collaboration and trust throughout the supply chain.However, such potential cannot be achieved without blockchain "oracles" needed to bridge the on-chain (i.e., blockchain system) and off-chain (i.e., real-life physical project) worlds.This study presents an innovative solution that exploits smart construction objects (SCOs).It develops a SCOs-enabled blockchain oracles (SCOs-BOs) framework.To instantiate this framework, the system architecture of a blockchain-enabled construction supply chain management (BCSCM) system is developed and validated using a case study, whereby four primary smart contracts are examined in the context of off-site logistics and on-site assembly services.The validation results show that accurate data is retrieved against malicious data in each request, and the corresponding reputation scores are successfully recorded.The innovativeness of the research lies in two aspects.In addition to mobilizing SCOs as blockchain oracles to bridge the on-chain and off-chain worlds, it develops a decentralized SCO network to avoid the single point of failure (SPoF) problem widely existing in blockchain systems.This study contributes to existing research and practice to harness the power of blockchain in construction.
Secure and Efficient Federated Learning for Smart Grid With Edge-Cloud Collaboration

With the prevalence of smart appliances, smart meters, and Internet of Things (IoT) devices in smart grids, artificial intelligence (AI) built on the rich IoT big data enables various energy data analysis applications and brings intelligent and personalized energy services for users. In conventional AI of Things (AIoT) paradigms, a wealth of individual energy data distributed across users’ IoT devices needs to be migrated to a central storage (e.g., cloud or edge device) for knowledge extraction, which may impose severe privacy violation and data misuse risks. Federated learning, as an appealing privacy-preserving AI paradigm, enables energy data owners (EDOs) to cooperatively train a shared AI model without revealing the local energy data. Nevertheless, potential security and efficiency concerns still impede the deployment of federated-learning-based AIoT services in smart grids due to the low-quality shared local models, non-independently and identically distributed (non-IID) data distributions, and unpredictable communication delays. In this article, we propose a secure and efficient federated-learning-enabled AIoT scheme for private energy data sharing in smart grids with edge-cloud collaboration. Specifically, we first introduce an edge-cloud-assisted federated learning framework for communication-efficient and privacy-preserving energy data sharing of users in smart grids. Then, by considering non-IID effects, we design a local data evaluation mechanism in federated learning and formulate two optimization problems for EDOs and energy service providers. Furthermore, due to the lack of knowledge of multidimensional user private information in practical scenarios, a two-layer deep reinforcement-learning-based incentive algorithm is developed to promote EDOs’ participation and high-quality model contribution. Extensive simulation results show that the proposed scheme can effectively stimulate EDOs to share high-quality local model updates and improve the communication efficiency.
A Latent Factor Analysis-Based Approach to Online Sparse Streaming Feature Selection

Online streaming feature selection (OSFS) has attracted extensive attention during the past decades. Current approaches commonly assume that the feature space of fixed data instances dynamically increases without any missing data. However, this assumption does not always hold in many real applications. Motivated by this observation, this study aims to implement online feature selection from sparse streaming features, i.e., features flow in one by one with missing data as instance count remains fixed. To do so, this study proposes a latent-factor-analysis-based online sparse-streaming-feature selection algorithm (LOSSA). Its main idea is to apply latent factor analysis to pre-estimate missing data in sparse streaming features before conducting feature selection, thereby addressing the missing data issue effectively and efficiently. Theoretical and empirical studies indicate that LOSSA can significantly improve the quality of OSFS when missing data are encountered in target instances.
Privacy-Enhanced Federated Learning Against Poisoning Adversaries

Federated learning (FL), as a distributed machine learning setting, has received considerable attention in recent years. To alleviate privacy concerns, FL essentially promises that multiple parties jointly train the model by exchanging gradients rather than raw data. However, intrinsic privacy issue still exists in FL, e.g., user’s training samples could be revealed by solely inferring gradients. Moreover, the emerging poisoning attack also poses a crucial security threat to FL. In particular, due to the distributed nature of FL, malicious users may submit crafted gradients during the training process to undermine the integrity and availability of the model. Furthermore, there exists a contradiction in simultaneously addressing two issues, that is, privacy-preserving FL solutions are dedicated to ensuring gradients indistinguishability, whereas the defenses against poisoning attacks tend to remove outliers based on their similarity. To solve such a dilemma, in this paper, we aim to build a bridge between the two issues. Specifically, we present a privacy-enhanced FL (PEFL) framework that adopts homomorphic encryption as the underlying technology and provides the server with a channel to punish poisoners via the effective gradient data extraction of the logarithmic function. To the best of our knowledge, the PEFL is the first effort to efficiently detect the poisoning behaviors in FL under ciphertext. Detailed theoretical analyses illustrate the security and convergence properties of the scheme. Moreover, the experiments conducted on real-world datasets show that the PEFL can effectively defend against label-flipping and backdoor attacks, two representative poisoning attacks in FL.
Improved breast cancer histological grading using deep learning

The Nottingham histological grade (NHG) is a well-established prognostic factor for breast cancer that is broadly used in clinical decision making. However, ∼50% of patients are classified as grade 2, an intermediate risk group with low clinical value. To improve risk stratification of NHG 2 breast cancer patients, we developed and validated a novel histological grade model (DeepGrade) based on digital whole-slide histopathology images (WSIs) and deep learning.In this observational retrospective study, routine WSIs stained with haematoxylin and eosin from 1567 patients were utilised for model optimisation and validation. Model generalisability was further evaluated in an external test set with 1262 patients. NHG 2 cases were stratified into two groups, DG2-high and DG2-low, and the prognostic value was assessed. The main outcome was recurrence-free survival.DeepGrade provides independent prognostic information for stratification of NHG 2 cases in the internal test set, where DG2-high showed an increased risk for recurrence (hazard ratio [HR] 2.94, 95% confidence interval [CI] 1.24-6.97, P = 0.015) compared with the DG2-low group after adjusting for established risk factors (independent test data). DG2-low also shared phenotypic similarities with NHG 1, and DG2-high with NHG 3, suggesting that the model identifies morphological patterns in NHG 2 that are associated with more aggressive tumours. The prognostic value of DeepGrade was further assessed in the external test set, confirming an increased risk for recurrence in DG2-high (HR 1.91, 95% CI 1.11-3.29, P = 0.019).The proposed model-based stratification of patients with NHG 2 tumours is prognostic and adds clinically relevant information over routine histological grading. The methodology offers a cost-effective alternative to molecular profiling to extract information relevant for clinical decisions.
Fake it till you make it: face analysis in the wild using synthetic data alone

We demonstrate that it is possible to perform face-related computer vision in the wild using synthetic data alone. The community has long enjoyed the benefits of synthesizing training data with graphics, but the domain gap between real and synthetic data has remained a problem, especially for human faces. Researchers have tried to bridge this gap with data mixing, domain adaptation, and domain-adversarial training, but we show that it is possible to synthesize data with minimal domain gap, so that models trained on synthetic data generalize to real in-the-wild datasets. We describe how to combine a procedurally-generated parametric 3D face model with a comprehensive library of hand-crafted assets to render training images with unprecedented realism and diversity. We train machine learning systems for face-related tasks such as landmark localization and face parsing, showing that synthetic data can both match real data in accuracy as well as open up new approaches where manual labeling would be impossible.
An Industrial IoT-Based Blockchain-Enabled Secure Searchable Encryption Approach for Healthcare Systems Using Neural Network

The IoT refers to the interconnection of things to the physical network that is embedded with software, sensors, and other devices to exchange information from one device to the other. The interconnection of devices means there is the possibility of challenges such as security, trustworthiness, reliability, confidentiality, and so on. To address these issues, we have proposed a novel group theory (GT)-based binary spring search (BSS) algorithm which consists of a hybrid deep neural network approach. The proposed approach effectively detects the intrusion within the IoT network. Initially, the privacy-preserving technology was implemented using a blockchain-based methodology. Security of patient health records (PHR) is the most critical aspect of cryptography over the Internet due to its value and importance, preferably in the Internet of Medical Things (IoMT). Search keywords access mechanism is one of the typical approaches used to access PHR from a database, but it is susceptible to various security vulnerabilities. Although blockchain-enabled healthcare systems provide security, it may lead to some loopholes in the existing state of the art. In literature, blockchain-enabled frameworks have been presented to resolve those issues. However, these methods have primarily focused on data storage and blockchain is used as a database. In this paper, blockchain as a distributed database is proposed with a homomorphic encryption technique to ensure a secure search and keywords-based access to the database. Additionally, the proposed approach provides a secure key revocation mechanism and updates various policies accordingly. As a result, a secure patient healthcare data access scheme is devised, which integrates blockchain and trust chain to fulfill the efficiency and security issues in the current schemes for sharing both types of digital healthcare data. Hence, our proposed approach provides more security, efficiency, and transparency with cost-effectiveness. We performed our simulations based on the blockchain-based tool Hyperledger Fabric and OrigionLab for analysis and evaluation. We compared our proposed results with the benchmark models, respectively. Our comparative analysis justifies that our proposed framework provides better security and searchable mechanism for the healthcare system.
Age-Invariant Face Recognition by Multi-Feature Fusionand Decomposition with Self-attention

Different from general face recognition, age-invariant face recognition (AIFR) aims at matching faces with a big age gap. Previous discriminative methods usually focus on decomposing facial feature into age-related and age-invariant components, which suffer from the loss of facial identity information. In this article, we propose a novel Multi-feature Fusion and Decomposition (MFD) framework for age-invariant face recognition, which learns more discriminative and robust features and reduces the intra-class variants. Specifically, we first sample multiple face images of different ages with the same identity as a face time sequence. Then, the multi-head attention is employed to capture contextual information from facial feature series, extracted by the backbone network. Next, we combine feature decomposition with fusion based on the face time sequence to ensure that the final age-independent features effectively represent the identity information of the face and have stronger robustness against the aging process. Besides, we also mitigate imbalanced age distribution in the training data by a re-weighted age loss. We experimented with the proposed MFD over the popular CACD and CACD-VS datasets, where we show that our approach improves the AIFR performance than previous state-of-the-art methods. We simultaneously show the performance of MFD on LFW dataset.
An object tracking framework with recapture based on correlation filters and Siamese networks

Recently, both correlation filters-based and Siamese-based trackers have achieved great progress in the field of visual object tracking. Whereas, some trackers perform not that well under some tough situations. Aimed at occlusions and background clutters, we propose a tracking framework combining correlation filter tracking and Siamese-based object tracking. First, we combine deep features with handcrafted features in correlation filter learning. Then, we propose a new robust criterion to evaluate the robustness of the tracking results and decide whether to start the Siamese tracking model according to the criterion. If the robustness evaluation value is lower than the adaptive threshold, we start the Siamese tracking for object recapture. We conduct the experiment to evaluate our tracker in four datasets to validate the effectiveness. The experimental results show that our method achieves state-of-the-art performance. It is worth mentioning that our tracker performs favorably against many existing trackers in the TC128 dataset.
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

Motivated by the success of masked language modeling (MLM) in pre-training natural language processing models, we propose w2v-BERT that explores MLM for self-supervised speech representation learning. w2v-BERT is a framework that combines contrastive learning and MLM, where the former trains the model to discretize input continuous speech signals into a finite set of discriminative speech tokens, and the latter trains the model to learn contextualized speech representations via solving a masked prediction task consuming the discretized tokens. In contrast to existing MLM-based speech pre-training frameworks such as HuBERT, which relies on an iterative re-clustering and re-training process, or vq-wav2vec, which concatenates two separately trained modules, w2v-BERT can be optimized in an end-to-end fashion by solving the two self-supervised tasks (the contrastive task and MLM) simultaneously. Our experiments show that w2v-BERT achieves competitive results compared to current state-of-the-art pre-trained models on the LibriSpeech benchmarks when using the Libri-Light 60k corpus as the unsupervised data. In particular, when compared to published models such as conformer-based wav2vec 2.0 and HuBERT, our model shows 5% to 10% relative WER reduction on the test-clean and test-other subsets. When applied to the Google's Voice Search traffic dataset, w2v-BERT outperforms our internal conformer-based wav2vec 2.0 by more than 30% relatively.
PyTorch

PyTorch is a library for Python programs that encourages deep learning programs. With this receptiveness and convenience found in (Deep Learning for Computer Vision: Expert techniques to train advanced neural networks using TensorFlow and Keras. [Authors: RajalingappaaShanmugamani]), PyTorch makes it useful in developing deep neural networks. It has an expansive scope and is applied for various applications. As Python is for programming, PyTorch is both a magnificent prologue to profound learning just as an instrument usable in proficient real-world applications.
Rateless Codes for Near-Perfect Load Balancing in Distributed Matrix-Vector Multiplication

Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes -- computing nodes that unpredictably slowdown or fail -- is a major bottleneck in such distributed computations. Ideal load balancing strategies that dynamically allocate more tasks to faster nodes require knowledge or monitoring of node speeds as well as the ability to quickly move data. Recently proposed fixed-rate erasure coding strategies can handle unpredictable node slowdown, but they ignore partial work done by straggling nodes thus resulting in a lot of redundant computation. We propose a rateless fountain coding strategy that achieves the best of both worlds -- we prove that its latency is asymptotically equal to ideal load balancing, and it performs asymptotically zero redundant computations. Our idea is to create linear combinations of the m rows of the matrix and assign these encoded rows to different worker nodes. The original matrix-vector product can be decoded as soon as slightly more than m row-vector products are collectively finished by the nodes. We conduct experiments in three computing environments: local parallel computing, Amazon EC2, and Amazon Lambda, which show that rateless coding gives as much as 3x speed-up over uncoded schemes.
Exploiting Color Volume and Color Difference for Salient Region Detection

Foreground and background cues can assist humans in quickly understanding visual scenes. In computer vision, however, it is difficult to detect salient objects when they touch the image boundary. Hence, detecting salient objects robustly under such circumstances without sacrificing precision and recall can be challenging. In this paper, we propose a novel model for salient region detection, namely, the foreground-center-background (FCB) saliency model. Its main highlights as follows. First, we use regional color volume as the foreground, together with perceptually uniform color differences within regions to detect salient regions. This can highlight salient objects robustly, even when they touched the image boundary, without greatly sacrificing precision and recall. Second, we employ center saliency to detect salient regions together with foreground and background cues, which improves saliency detection performance. Finally, we propose a novel and simple yet efficient method that combines foreground, center, and background saliency. Experimental validation with three well-known benchmark data sets indicates that the FCB model outperforms several state-of-the-art methods in terms of precision, recall, F-measure, and particularly, the mean absolute error. Salient regions are brighter than those of some existing state-of-the-art methods.