Reviewerly/exHarmony huggingface dataset:
Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information

Object detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequently, neither of these two data sources yields satisfactory results when used alone. While some scholars have combined visible and thermal images for object detection, most did not consider the illumination conditions and the different contributions of diverse data sources to the results. In addition, few studies have made use of the temperature characteristics of thermal images. Therefore, in the present study, visible and thermal images are utilized as the dataset, and RetinaNet is used as the baseline to fuse features from different data sources for object detection. Moreover, a dynamic weight fusion method, which is based on channel attention according to different illumination conditions, is used in the fusion component, and the channel attention and a priori temperature mask (CAPTM) module is proposed; the CAPTM can be applied to a deep learning network as a priori knowledge and maximizes the advantage of temperature information from thermal images. The main innovations of the present research include the following: (1) the consideration of different illumination conditions and the use of different fusion parameters for different conditions in the feature fusion of visible and thermal images; (2) the dynamic fusion of different data sources in the feature fusion of visible and thermal images; (3) the use of temperature information as a priori knowledge (CAPTM) in feature extraction. To a certain extent, the proposed methods improve the accuracy of object detection at night or under other weak illumination conditions and with a single data source. Compared with the state-of-the-art (SOTA) method, the proposed method is found to achieve superior detection accuracy with an overall mean average precision (mAP) improvement of 0.69%, including an AP improvement of 2.55% for the detection of the Person category. The results demonstrate the effectiveness of the research methods for object detection, especially temperature information-rich object detection.
The impact of platform restriction on manufacturer quality transparency in the blockchain era

This study investigates a supply chain comprising of a manufacturer and a retail platform, wherein the former decides the quality information transparency strategy of products and the latter determines whether or not to implement price parity clauses (PPCs). The manufacturer can directly sell the end-of-season product to the end-consumer market and indirectly via the retail platform. Specifically, this study considers two quality information transparency strategies (i.e. ex-ante blockchain-adoption and ex-post voluntary disclosure) that differ on whether the manufacturer’s decision on quality information transparency strategy is made before or after observing the actual product quality. Results show that the manufacturer may prefer the ex-ante blockchain-adoption strategy in specific conditions but maintains the flexibility of information disclosure under ex-post voluntary disclosure strategy. Moreover, the implementation of PPCs significantly reduces the manufacturer’s incentive to disclose a large extent of quality information transparency when the effort cost is high or the service effect coefficient is low. By contrast, the manufacturer is more likely to adopt information transparency strategies under PPCs when the effort cost is low and the service effect coefficient is high. More interestingly, contrary to theories of harm from PPCs, the implementation of PPCs may simultaneously benefit the manufacturer and retail platform, achieving a ‘win-win' situation.
Broad Learning Based Dynamic Fuzzy Inference System With Adaptive Structure and Interpretable Fuzzy Rules

This article investigates the feasibility of applying the broad learning system (BLS) to realize a novel Takagi–Sugeno–Kang (TSK) neuro-fuzzy model, namely a broad learning based dynamic fuzzy inference system (BL-DFIS). It not only improves the accuracy and interpretability of neuro-fuzzy models but also solves the challenging problem that models are incapable of determining the optimal architecture autonomously. BL-DFIS first accomplishes a TSK fuzzy system under the framework of BLS, in which an extreme learning machine auto-encoder is employed to obtain feature representation in a fast and analytical way, and an interpretable linguistic fuzzy rule is integrated into the enhancement node to ensure the high interpretability of the system. Meanwhile, the extended-enhancement unit is designed to achieve the first-order TSK fuzzy system. In addition, a dynamic incremental learning algorithm with internal pruning and updating mechanism is developed for the learning of BL-DFIS, which enables the system to automatically assemble the optimal structure to obtain a compact rule base and an excellent classification performance. Experiments on benchmark datasets demonstrate that the proposed BL-DFIS can achieve a better classification performance than some state-of-the-art nonfuzzy and neuro-fuzzy methods, simultaneously using the most parsimonious model structure.
Boosting the Decision-Making in Smart Ports by Using Blockchain

Blockchain technology (BC) offers an innovation platform for decentralized and transparent transactions in the maritime port industry. This technology allows guaranteeing trust, transparency and traceability of cargo and data to be tracked. Today, in the port systems of emerging market countries, BC technology is increasingly being incorporated into information and communication processes. In parallel, the social domain has begun to be explored due to the lack of link between the port and the city it occupies, and the need to incorporate public actors in decision-making at the governance level. In this sense, the present work aims to promote BC technology in order to transform data and information into useful knowledge for effective decision making, through the use of Crowdsourcing. A Crowdsourcing Blockchain (CrowdBC) conceptual framework and its architecture are generated for a port system in which the cyber-technological, social and cognitive domains (CSTC) of smart ports, the knowledge generation process and Crowdsourcing technology are interrelated. Finally, opportunities are discussed for ports that are in permanent development to reduce the gaps with the smart industry. As a discussion, two possible scenarios and recommendations for future implementations that consider the social and cognitive aspects of Industry 4.0 are presented.
Deep Dual Support Vector Data description for anomaly detection on attributed networks

Networks are ubiquitous in the real world such as social networks and communication networks, and anomaly detection on networks aims at finding nodes whose structural or attributed patterns deviate significantly from the majority of reference nodes. However, most of the traditional anomaly detection methods neglect the relation structure information among data points and therefore cannot effectively generalize to the graph structure data. In this paper, we propose an end-to-end model of Deep Dual Support Vector Data description based Autoencoder (Dual-SVDAE) for anomaly detection on attributed networks, which considers both the structure and attribute for attributed networks. Specifically, Dual-SVDAE consists of a structure autoencoder and an attribute autoencoder to learn the latent representation of the node in the structure space and attribute space, respectively. Then, a dual-hypersphere learning mechanism is imposed on them to learn two hyperspheres of normal nodes from the structure and attribute perspectives, respectively. Moreover, to achieve joint learning between the structure and attribute of the network, we fuse the structure embedding and attribute embedding as the final input of the feature decoder to generate the node attribute. Finally, abnormal nodes can be detected by measuring the distance of nodes to the learned center of each hypersphere in the latent structure space and attribute space, respectively. Extensive experiments on the real-world attributed networks show that Dual-SVDAE consistently outperforms the state-of-the-arts, which demonstrates the effectiveness of the proposed method.
Multiscale parallel deep CNN (mpdCNN) architecture for the real low-resolution face recognition for surveillance

Images from the surveillance networks are being extensively used for the purpose of monitoring and criminal investigations. However, it is often difficult to recognize faces using the data from the surveillance networks because the resolution of the face in the images captured is too low. Further, the low-resolution images have varying magnitude of facial feature content because of wide variations in illumination, pose, resolution and the distance from which the image is captured. Also, a single face recognition solution is not able to recognize faces efficiently in both high and low-resolution images. Wide variations in facial feature content in high and low-resolution images causes difficulty in classification of the features by a single model for the purpose of face recognition. We present a Deep-CNN based architecture called mpdCNN to solve the problem of face recognition in low as well as high-resolution images with high accuracy and robustness. Our proposed architecture mpdCNN gives 88.6% accuracy on the SCface database which is an impressive improvement over the state-of-the-art algorithms. We also achieved an accuracy of above 99% on normal to high-resolution databases for face recognition.
Camera style transformation with preserved self-similarity and domain-dissimilarity in unsupervised person re-identification

The inconsistency caused by different factors, such as different camera imaging methods, complex imaging environments, and changes in light, present a huge challenge to person re-identification (re-ID). Unsupervised domain adaptation (UDA) can solve the inconsistency issue to a certain extent, but different datasets may not have any overlapping of people's identities. Therefore, it is necessary to pay attention to people's identities in solving domain-dissimilarity. A camera imaging style transformation with preserved self-similarity and domain-dissimilarity (CSPSD) is proposed to solve the cross-domain issue in person re-ID. First, CycleGAN is applied to determine the style conversion between source and target domains. Intra-domain identity constraints are used to maintain identity consistency between source and target domains during the image style transformation process. Maximum mean difference (MMD) is used to reduce the difference in feature distribution between source and target domains. Then, a one-to-n mapping method is proposed to achieve the mapping between positive pairs and distinguish negative pairs. Any sample image from the source domain and its transformed image or a transformed image with the same identity information compose a positive pair. The transformed image and any image from the target domain compose a negative pair. Next, a circle loss function is used to improve the learning speed of positive and negative pairs. Finally, the proposed CSPSD that can effectively reduce the difference between domains and an existing feature learning network work together to learn a person re-ID model. The proposed method is applied to three public datasets, Market-1501, DukeMTMC-reID, and MSMT17. The comparative experimental results confirm the proposed method can achieve highly competitive recognition accuracy in person re-ID.
Newton-interpolation-based zk-SNARK for Artificial Internet of Things

Artificial Internet of Things (AIoT) is that the system collects all kinds of information in real-time through various sensors, and intelligence analysis of the data through machine learning in the terminal equipment, edge domains, or cloud centers, including positioning, comparison, forecasting, scheduling, etc. which brings about the data security and privacy issues. The blockchain is a tamper-evident, unforgeable distributed ledger that protects security and privacy through the famous algorithm zk-SNARK, which is also widely used in virtual digital currencies such as Zcash. In addition, by using zk-SNARK technology in the Loopring DEX 3.0 in Ethereum, not only decentralization but also transaction performance can be guaranteed. However, there are three main problems of zk-SNARK, one is the need to guarantee calculation accuracy, two is the long time to generate evidence, especially when using Lagrangian interpolation to QAP the transaction data requires more computation; the last is the poor scalability, especially when nodes need to recalculate all data when adding new transactions. In this paper, we propose a modified zk-SNARK based on Newtonian interpolation, improve the QAP part of zk-SNARK by Newtonian interpolation, and verify the correctness of the scheme through instantiation. Finally, we analyze the computational efficiency of the two interpolation methods, and the results show that Newton interpolation solves the above two problems in the original zk-SNARK, and significantly reduces the time complexity of the algorithm, which can further promote the application of blockchain in data management of AIoT.
COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation

There has been a recent surge of interest in cross-modal pre-training. However, existed approaches pre-train a one-stream model to learn joint vision-language representation, which suffers from calculation explosion when conducting cross-modal retrieval. In this work, we propose the Contrastive Cross-Modal Knowledge Sharing Pretraining (COOKIE) method to learn universal text-image representations. There are two key designs in it, one is the weight-sharing transformer on top of the visual and textual encoders to align text and image semantically, the other is three kinds of contrastive learning designed for sharing knowledge between different modalities. Cross-modal knowledge sharing greatly promotes the learning of unimodal representation. Experiments on multi-modal matching tasks including cross-modal retrieval, text matching, and image retrieval show the effectiveness and efficiency of our pre-training framework. Our COOKIE finetuned on cross-modal datasets MSCOCO, Flickr30K, and MSRVTT achieves new state-of-the-art results while using only 3/1000 inference time comparing to one-stream models. There are also 5.7% and 3.9% improvements in the task of image retrieval and text matching. Source code will be available at https://github.com/kywen1119/COOKIE.
Blockchain smart contract reference framework and program logic architecture for transactive energy systems

This paper proposes a reference framework for a transactive energy market based on distributed ledger technology such as blockchain. The framework design was based on the engineering requirements of a distribution-scale market, including participant needs, expected market transactions, and the cybersecurity constructs required to support a fair, secure, and efficient market operation. It leverages the existing blockchain attributes to provide clear value propositions that are applicable to transactive energy market applications. Those attributes are evaluated to not only facilitate the energy market mechanisms but also assist with the cybersecurity requirements, such as access control management (including identification, authentication, and authorization); data security and integrity; and resilience management (decentralization, scalability, and performance during faults). The validity of the proposed framework is demonstrated using a real-time, five-minute double-auction market. The results highlight the framework benefits and provide a strong validation of its applicability to blockchain-based transactive energy systems. The proposed design can help to support machine-to-machine transactions that can be directly integrated into complex grid operations, such as automated market bidding and self-sufficient, resilience-capable grids.
SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentation

Unlike conventional zero-shot classification, zero-shot semantic segmentation predicts a class label at the pixel level instead of the image level. When solving zero-shot semantic segmentation problems, the need for pixel-level prediction with surrounding context motivates us to incorporate spatial information using positional encoding. We improve standard positional encoding by introducing the concept of Relative Positional Encoding, which integrates spatial information at the feature level and can handle arbitrary image sizes. Furthermore, while self-training is widely used in zero-shot semantic segmentation to generate pseudo-labels, we propose a new knowledge-distillation-inspired self-training strategy, namely Annealed Self-Training, which can automatically assign different importance to pseudo-labels to improve performance. We systematically study the proposed Relative Positional Encoding and Annealed Self-Training in a comprehensive experimental evaluation, and our empirical results confirm the effectiveness of our method on three benchmark datasets.
Explainable Machine Learning for Fraud Detection

The application of machine learning to support the processing of large data sets holds promise in many industries. We explore explainability methods in the domain of real-time fraud detection by investigating the selection of appropriate background data sets and runtime tradeoffs on supervised and unsupervised models.
Joint Visual and Audio Learning for Video Highlight Detection

In video highlight detection, the goal is to identify the interesting moments within an unedited video. Although the audio component of the video provides important cues for highlight detection, the majority of existing efforts focus almost exclusively on the visual component. In this paper, we argue that both audio and visual components of a video should be modeled jointly to retrieve its best moments. To this end, we propose an audio-visual network for video highlight detection. At the core of our approach lies a bimodal attention mechanism, which captures the interaction between the audio and visual components of a video, and produces fused representations to facilitate highlight detection. Furthermore, we introduce a noise sentinel technique to adaptively discount a noisy visual or audio modality. Empirical evaluations on two benchmark datasets demonstrate the superior performance of our approach over the state-of-the-art methods.
LabOR: Labeling Only if Required for Domain Adaptive Semantic Segmentation

Unsupervised Domain Adaptation (UDA) for semantic segmentation has been actively studied to mitigate the domain gap between label-rich source data and unlabeled target data. Despite these efforts, UDA still has a long way to go to reach the fully supervised performance. To this end, we propose a Labeling Only if Required strategy, LabOR, where we introduce a human-in-the-loop approach to adaptively give scarce labels to points that a UDA model is uncertain about. In order to find the uncertain points, we generate an inconsistency mask using the proposed adaptive pixel selector and we label these segment-based regions to achieve near supervised performance with only a small fraction (about 2.2%) ground truth points, which we call "Segment based Pixel-Labeling (SPL)." To further reduce the efforts of the human annotator, we also propose "Point based Pixel-Labeling (PPL)," which finds the most representative points for labeling within the generated inconsistency mask. This reduces efforts from 2.2% segment label → 40 points label while minimizing performance degradation. Through extensive experimentation, we show the advantages of this new framework for domain adaptive semantic segmentation while minimizing human labor costs.
Multi-view Spectral Clustering with Adaptive Graph Learning and Tensor Schatten p-norm

Although the promising clustering performance, existing graph-based multi-view clustering methods still have the following deficiencies. (1) They cannot well exploit the salient difference between graphs of different views, resulting in suboptimal graphs. (2) They minimize the divergence between graphs of different views by one-dimensional, element by element metric model. Thus, they cannot well exploit the complementary information embedded in graphs. (3) They fail to simultaneously take local and global intrinsic geometric structures into account, resulting in suboptimal clustering performance. To handle the aforementioned problems, we propose Multi-view Spectral Clustering with Adaptive Graph Learning and Tensor Schatten p-norm. Specifically, we present an adaptive weighted strategy that directly takes into account the contribution among different views for clustering. To well learn a good graph, which well characterizes cluster structure, we integrate local and global structure learning into a unified framework and leverage tensor Schatten p-norm regularizer to minimize the divergence between graphs of different views. Extensive experiments on benchmark datasets have demonstrated the superiority of the proposed method.
Social Distance Monitoring Approach Using Wearable Smart Tags

Coronavirus has affected millions of people worldwide, with the rate of infected people still increasing. The virus is transmitted between people through direct, indirect, or close contact with infected people. To help prevent the social transmission of COVID-19, this paper presents a new smart social distance system that allows individuals to keep social distances between others in indoor and outdoor environments, avoiding exposure to COVID-19 and slowing its spread locally and across the country. The proposed smart monitoring system consists of a new smart wearable prototype of a compact and low-cost electronic device, based on human detection and proximity distance functions, to estimate the social distance between people and issue a notification when the social distance is less than a predefined threshold value. The developed social system has been validated through several experiments, and achieved a high acceptance rate (96.1%) and low localization error (&lt;6 m).
GistNet: a Geometric Structure Transfer Network for Long-Tailed Recognition

The problem of long-tailed recognition, where the number of examples per class is highly unbalanced, is considered. It is hypothesized that the well known tendency of standard classifier training to overfit to popular classes can be exploited for effective transfer learning. Rather than eliminating this overfitting, e.g. by adopting popular class-balanced sampling methods, the learning algorithm should instead leverage this overfitting to transfer geometric information from popular to low-shot classes. A new classifier architecture, GistNet, is proposed to support this goal, using constellations of classifier parameters to encode the class geometry. A new learning algorithm is then proposed for GeometrIc Structure Transfer (GIST), with resort to a combination of loss functions that combine class-balanced and random sampling to guarantee that, while overfitting to the popular classes is restricted to geometric parameters, it is leveraged to transfer class geometry from popular to few-shot classes. This enables better generalization for few-shot classes without the need for the manual specification of class weights, or even the explicit grouping of classes into different types. Experiments on two popular long-tailed recognition datasets show that GistNet outperforms existing solutions to this problem.