Reviewerly/exHarmony huggingface dataset:
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation

Supervised learning in large discriminative models is a mainstay for modern computer vision. Such an approach necessitates investing in large-scale human-annotated datasets for achieving state-of-the-art results. In turn, the efficacy of supervised learning may be limited by the size of the human annotated dataset. This limitation is particularly notable for image segmentation tasks, where the expense of human annotation is especially large, yet large amounts of unlabeled data may exist. In this work, we ask if we may leverage semi-supervised learning in unlabeled video sequences and extra images to improve the performance on urban scene segmentation, simultaneously tackling semantic, instance, and panoptic segmentation. The goal of this work is to avoid the construction of sophisticated, learned architectures specific to label propagation (e.g., patch matching and optical flow). Instead, we simply predict pseudo-labels for the unlabeled data and train subsequent models with both human-annotated and pseudo-labeled data. The procedure is iterated for several times. As a result, our Naive-Student model, trained with such simple yet effective iterative semi-supervised learning, attains state-of-the-art results at all three Cityscapes benchmarks, reaching the performance of 67.8% PQ, 42.6% AP, and 85.2% mIOU on the test set. We view this work as a notable step towards building a simple procedure to harness unlabeled video sequences and extra images to surpass state-of-the-art performance on core computer vision tasks.
SMR: Medical Knowledge Graph Embedding for Safe Medicine Recommendation

Most of the existing medicine recommendation systems that are mainly based on electronic medical records (EMRs) are significantly assisting doctors to make better clinical decisions benefiting both patients and caregivers. Even though the growth of EMRs is at a lighting fast speed in the era of big data, content limitations in EMRs restrain the existed recommendation systems to reflect relevant medical facts, such as drug-drug interactions. Many medical knowledge graphs that contain drug-related information, such as DrugBank, may give hope for the recommendation systems. However, the direct use of these knowledge graphs in systems suffers from robustness caused by the incompleteness of the graphs. To address these challenges, we stand on recent advances in graph embedding learning techniques and propose a novel framework, called Safe Medicine Recommendation (SMR), in this paper. Specifically, SMR first constructs a high-quality heterogeneous graph by bridging EMRs (MIMIC-III) and medical knowledge graphs (ICD-9 ontology and DrugBank). Then, SMR jointly embeds diseases, medicines, patients, and their corresponding relations into a shared lower dimensional space. Finally, SMR uses the embeddings to decompose the medicine recommendation into a link prediction process while considering the patient's diagnoses and adverse drug reactions. Extensive experiments on real datasets are conducted to evaluate the effectiveness of proposed framework.
Artificial intelligence in E-commerce fulfillment: A case study of resource orchestration at Alibaba’s Smart Warehouse

Despite heightened interest, integrating artificial intelligence (AI) into businesses remains challenging. Recent surveys show that up to 85 % of AI initiatives ultimately fail to deliver on their promises. Studies on successful AI applications that could provide invaluable lessons for organizations embarking on their AI journey are still lacking. Therefore, this study aims to understand how AI technology, people, and processes should be managed to successfully create value. Building on the resource orchestration perspective, this study analyzes the successful applications of AI at Alibaba's e-commerce fulfillment center. The findings indicate that the key AI resources include data, AI algorithms, and robots. These resources must be orchestrated (e.g., coordinated, leveraged, deployed) to work with other related resources, such as warehouse facilities and existing information systems, to generate strong AI capabilities. The key AI capabilities generated include forecasting, planning, and learning. More importantly, AI capabilities are not independent – they interact and coevolve with human capabilities to create business value in terms of efficiency (e.g., space optimization, labor productivity) and effectiveness (e.g., error reduction). The implications of understanding these social informatics of AI for research and practice are discussed.
From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

Machine learning (ML) tasks are becoming ubiquitous in today's network applications. Federated learning has emerged recently as a technique for training ML models at the network edge by leveraging processing capabilities across the nodes that collect the data. There are several challenges with employing conventional federated learning in contemporary networks, due to the significant heterogeneity in compute and communication capabilities that exist across devices. To address this, we advocate a new learning paradigm called fog learning, which will intelligently distribute ML model training across the continuum of nodes from edge devices to cloud servers. Fog learning enhances federated learning along three major dimensions: network, heterogeneity, and proximity. It considers a multi-layer hybrid learning framework consisting of heterogeneous devices with various proximities. It accounts for the topology structures of the local networks among the heterogeneous nodes at each network layer, orchestrating them for collaborative/cooperative learning through device-to-device communications. This migrates from star network topologies used for parameter transfers in federated learning to more distributed topologies at scale. We discuss several open research directions toward realizing fog learning.
Video Face Manipulation Detection Through Ensemble of CNNs

In the last few years, several techniques for facial manipulation in videos have been successfully developed and made available to the masses (i.e., FaceSwap, deepfake, etc.). These methods enable anyone to easily edit faces in video sequences with incredibly realistic results and a very little effort. Despite the usefulness of these tools in many fields, if used maliciously, they can have a significantly bad impact on society (e.g., fake news spreading, cyber bullying through fake revenge porn). The ability of objectively detecting whether a face has been manipulated in a video sequence is then a task of utmost importance. In this paper, we tackle the problem of face manipulation detection in video sequences targeting modern facial manipulation techniques. In particular, we study the ensembling of different trained Convolutional Neural Network (CNN) models. In the proposed solution, different models are obtained starting from a base network (i.e., EfficientNetB4) making use of two different concepts: (i) attention layers; (ii) siamese training. We show that combining these networks leads to promising face manipulation detection results on two publicly available datasets with more than 119000 videos.
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks

Deep learning (DL) techniques are gaining more and more attention in the software engineering community. They have been used to support several code-related tasks, such as automatic bug fixing and code comments generation. Recent studies in the Natural Language Processing (NLP) field have shown that the Text-To-Text Transfer Transformer (T5) architecture can achieve state-of-the-art performance for a variety of NLP tasks. The basic idea behind T5 is to first pre-train a model on a large and generic dataset using a self-supervised task (e.g., filling masked words in sentences). Once the model is pre-trained, it is fine-tuned on smaller and specialized datasets, each one related to a specific task (e.g., language translation, sentence classification). In this paper, we empirically investigate how the T5 model performs when pre-trained and fine-tuned to support code-related tasks. We pre-train a T5 model on a dataset composed of natural language English text and source code. Then, we fine-tune such a model by reusing datasets used in four previous works that used DL techniques to: (i) fix bugs, (ii) inject code mutants, (iii) generate assert statements, and (iv) generate code comments. We compared the performance of this single model with the results reported in the four original papers proposing DL-based solutions for those four tasks. We show that our T5 model, exploiting additional data for the self-supervised pre-training phase, can achieve performance improvements over the four baselines.
Blockchain and renewable energy: Integration challenges in circular economy era

Renewable energy technologies play a crucial role in reducing the energy consumption by exploiting natural energy resources. With the increase in the trend towards the use of renewable energy resources energy distribution networks have become more complex. As such, it is envisioned that efficient distribution of the generated energy, illegal energy use, unfair pricing, and individual energy producers' entry into the market will be the key issues that need be tackled in near future. In this study, we discuss in order to eliminate the problems experienced in the energy grid management process, the blockchain concept, and its integration with renewable energy systems. First, we develop a novel integration process of blockchain with the renewable energy systems under the circular economy perspective in order to ensure the sustainability of energy grid management systems, followed by discussions on the advantages of the proposed integration process for energy policy makers. Second, the challenges of blockchain faced during the integration to a circular economy are presented. In order to prioritize the challenges encountered in the integration process, a numerical analysis is conducted using the Pythagorean Fuzzy Analytical Hierarchy Process method based on expert opinions, and corresponding managerial implications are included.
Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding

Anomaly detection is a crucial task for monitoring various status (i.e., metrics) of entities (e.g., manufacturing systems and Internet services), which are often characterized by multivariate time series (MTS). In practice, it's important to precisely detect the anomalies, as well as to interpret the detected anomalies through localizing a group of most anomalous metrics, to further assist the failure troubleshooting. In this paper, we propose InterFusion, an unsupervised method that simultaneously models the inter-metric and temporal dependency for MTS. Its core idea is to model the normal patterns inside MTS data through hierarchical Variational AutoEncoder with two stochastic latent variables, each of which learns low-dimensional inter-metric or temporal embeddings. Furthermore, we propose an MCMC-based method to obtain reasonable embeddings and reconstructions at anomalous parts for MTS anomaly interpretation. Our evaluation experiments are conducted on four real-world datasets from different industrial domains (three existing and one newly published dataset collected through our pilot deployment of InterFusion). InterFusion achieves an average anomaly detection F1-Score higher than 0.94 and anomaly interpretation performance of 0.87, significantly outperforming recent state-of-the-art MTS anomaly detection methods.
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation

We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the weights at each decoder block vary spatially. For this purpose, we design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features, a multi-headed weight generating module which generates the weights of each block in the decoder immediately before they are consumed, for efficient memory utilization, and a primary network that is composed of novel dynamic patch-wise convolutions. Despite the usage of less-conventional blocks, our architecture obtains real-time performance. In terms of the runtime vs. accuracy trade-off, we surpass state of the art (SotA) results on popular semantic segmentation benchmarks: PASCAL VOC 2012 (val. set) and real-time semantic segmentation on Cityscapes, and CamVid. The code is available: https://nirkin.com/hyperseg.
An introduction to deep generative modeling

Abstract Deep generative models (DGM) are neural networks with many hidden layers trained to approximate complicated, high‐dimensional probability distributions using samples. When trained successfully, we can use the DGM to estimate the likelihood of each observation and to create new samples from the underlying distribution. Developing DGMs has become one of the most hotly researched fields in artificial intelligence in recent years. The literature on DGMs has become vast and is growing rapidly. Some advances have even reached the public sphere, for example, the recent successes in generating realistic‐looking images, voices, or movies; so‐called deep fakes. Despite these successes, several mathematical and practical issues limit the broader use of DGMs: given a specific dataset, it remains challenging to design and train a DGM and even more challenging to find out why a particular model is or is not effective. To help advance the theoretical understanding of DGMs, we introduce DGMs and provide a concise mathematical framework for modeling the three most popular approaches: normalizing flows, variational autoencoders, and generative adversarial networks. We illustrate the advantages and disadvantages of these basic approaches using numerical experiments. Our goal is to enable and motivate the reader to contribute to this proliferating research area. Our presentation also emphasizes relations between generative modeling and optimal transport.
Neural Feature Search for RGB-Infrared Person Re-Identification

RGB-Infrared person re-identification (RGB-IR ReID) is a challenging cross-modality retrieval problem, which aims at matching the person-of-interest over visible and infrared camera views. Most existing works achieve performance gains through manually-designed feature selection modules, which often require significant domain knowledge and rich experience. In this paper, we study a general paradigm, termed Neural Feature Search (NFS), to automate the process of feature selection. Specifically, NFS combines a dual-level feature search space and a differentiable search strategy to jointly select identity-related cues in coarse-grained channels and fine-grained spatial pixels. This combination allows NFS to adaptively filter background noises and concentrate on informative parts of human bodies in a data-driven manner. Moreover, a cross-modality contrastive optimization scheme further guides NFS to search features that can minimize modality discrepancy whilst maximizing inter-class distance. Extensive experiments on mainstream benchmarks demonstrate that our method outperforms state-of-the-arts, especially achieving better performance on the RegDB dataset with significant improvement of 11.20% and 8.64% in Rank-1 and mAP, respectively.
Ultra-High-Definition Image Dehazing via Multi-Guided Bilateral Learning

Convolutional neural networks (CNNs) have achieved significant success in the single image dehazing task. Unfortunately, most existing deep dehazing models have high computational complexity, which hinders their application to high-resolution images, especially for UHD (ultra-high-definition) or 4K resolution images. To address the problem, we propose a novel network capable of real-time dehazing of 4K images on a single GPU, which consists of three deep CNNs. The first CNN extracts haze-relevant features at a reduced resolution of the hazy input and then fits locally-affine models in the bilateral space. Another CNN is used to learn multiple full-resolution guidance maps corresponding to the learned bilateral model. As a result, the feature maps with high-frequency can be reconstructed by multi-guided bilateral upsampling. Finally, the third CNN fuses the high-quality feature maps into a dehazed image. In addition, we create a large-scale 4K image dehazing dataset to support the training and testing of compared models. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art dehazing approaches on various benchmarks.
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation

Video instance segmentation (VIS) is a new and critical task in computer vision. To date, top-performing VIS methods extend the two-stage Mask R-CNN by adding a tracking branch, leaving plenty of room for improvement. In contrast, we approach the VIS task from a new perspective and propose a one-stage spatial granularity network (SG-Net). Compared to the conventional two-stage methods, SG-Net demonstrates four advantages: 1) Our method has a one-stage compact architecture and each task head (detection, segmentation, and tracking) is crafted interdependently so they can effectively share features and enjoy the joint optimization; 2) Our mask prediction is dynamically performed on the sub-regions of each detected instance, leading to high-quality masks of fine granularity; 3) Each of our task predictions avoids using expensive proposal-based RoI features, resulting in much reduced runtime complexity per instance; 4) Our tracking head models objects’ centerness movements for tracking, which effectively enhances the tracking robustness to different object appearances. In evaluation, we present state-of-the-art comparisons on the YouTube-VIS dataset. Extensive experiments demonstrate that our compact one-stage method can achieve improved performance in both accuracy and inference speed. We hope our SG-Net could serve as a strong and flexible base-line for the VIS task. Our code will be available here <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks

Recently, convolutional neural networks (CNNs) have achieved great success in the field of artificial intelligence, including speech recognition, image recognition, and natural language processing. CNN architecture plays a key role in CNNs' performance. Most previous CNN architectures are hand-crafted, which requires designers to have rich expert domain knowledge. The trial-and-error process consumes a lot of time and computing resources. To solve this problem, researchers proposed the neural architecture search, which searches CNN architecture automatically, to satisfy different requirements. However, the blindness of the search strategy causes a 'loss of experience' in the early stage of the search process, and ultimately affects the results of the later stage. In this paper, we propose a self-adaptive mutation neural architecture search algorithm based on ResNet blocks and DenseNet blocks. The self-adaptive mutation strategy makes the algorithm adaptively adjust the mutation strategies during the evolution process to achieve better exploration. In addition, the whole search process is fully automatic, and users do not need expert knowledge about CNNs architecture design. In this paper, the proposed algorithm is compared with 17 state-of-the-art algorithms, including manually designed CNN and automatic search algorithms on CIFAR10 and CIFAR100. The results indicate that the proposed algorithm outperforms the competitors in terms of classification performance and consumes fewer computing resources.
An improved opposition-based marine predators algorithm for global optimization and multilevel thresholding image segmentation

A recent meta-heuristic algorithm called Marine Predators Algorithm (MPA) is enhanced using Opposition-Based Learning (OBL) termed MPA-OBL to improve their search efficiency and convergence. A comprehensive set of experiments are performed to evaluate the MPA-OBL and prove the impact influence of merging OBL strategy with the original MPA in enhancing the quality of the solutions and the acceleration of the convergence speed, using IEEE CEC’2020 benchmark problems as recently complex optimization benchmark. In order to evaluate the performance of the proposed MPA-OBL, the effectiveness of conjunction of OBL with the original MPA and the other counterparts are calculated and compared with LSHADE with semi-parameter adaptation hybrid with CMA-ES (LSHADE_SPACMA-OBL), Restart covariance matrix adaptation ES (CMA_ES-OBL), Differential evolution (DE-OBL), Harris hawk optimization (HHO-OBL), Sine cosine algorithm (SCA-OBL), Salp swarm algorithm (SSA-OBL), and the original MPA. The extensive results and comparisons in terms of optimization metrics have revealed the superiority of the proposed MPA-OBL in solving the IEEE CEC’2020 benchmark problems and improving the convergence speed. Moreover, as a sequel to the proposed MPA-OBL, also, we have conducted experiments using two objective functions of Otsu and Kapur’s methods over a variety of benchmark images at different level of thresholds based on three commonly evaluation matrices namely Peak signal-to-noise ratio (PSNR), Structural similarity (SSIM), and Feature similarity (FSIM) indices are analyzed qualitatively and quantitatively. Eventually, the statistical post-hoc analysis reveal that the MPA-OBL obtains highly efficient and reliable results in comparison with the other competitor algorithms.
Blockchain-Enabled IoT-BIM Platform for Supply Chain Management in Modular Construction

Configuring a trustworthy Internet of Things (IoT)–enabled building information modeling (BIM) platform (IBP) is significant for modular construction to ensure transparency, traceability, and immutability throughout its fragmented supply chain management. However, most current IBPs are designed adopting a centralized system architecture, which fails to achieve a decentralized and effective one to ensure a single point of truth in BIM and prevent a single point of failure in IoT networks. To address this challenge, this study introduces permissioned blockchain with IBP and proposes a novel service-oriented system architecture of blockchain-enabled IoT-BIM platform (BIBP) for the data-information-knowledge (DIK)–driven supply chain management in modular construction. First, infrastructure as a service (IaaS) is designed with hardware, core technologies, and protocols to offer accurate data from daily practice to blockchain BIM. Blockchain BIM as a service (BaaS) is then developed within the permissioned blockchain to ease the interoperability of the information, semantics, and meaningful inferences. Furthermore, software as a service (SaaS) is configured with decentralized applications to achieve knowledgeable operations or processes with a crash fault-tolerant consensus mechanism. The demonstrative case study in a modular student residence project evaluates the proposed BIBP system prototype with the performance analysis of storage cost, throughput, latency, privacy, and feedback from stakeholders. The results indicate that BIBP has an effective system architecture with acceptable throughput and latency, can save storage costs to achieve a single point of truth in BIM, and avoid a single point of failure for IoT networks with privacy and security-preserving mechanisms.