Reviewerly/exHarmony huggingface dataset:
Deep learning in multi-object detection and tracking: state of the art

Object detection and tracking is one of the most important and challenging branches in computer vision, and have been widely applied in various fields, such as health-care monitoring, autonomous driving, anomaly detection, and so on. With the rapid development of deep learning (DL) networks and GPU’s computing power, the performance of object detectors and trackers has been greatly improved. To understand the main development status of object detection and tracking pipeline thoroughly, in this survey, we have critically analyzed the existing DL network-based methods of object detection and tracking and described various benchmark datasets. This includes the recent development in granulated DL models. Primarily, we have provided a comprehensive overview of a variety of both generic object detection and specific object detection models. We have enlisted various comparative results for obtaining the best detector, tracker, and their combination. Moreover, we have listed the traditional and new applications of object detection and tracking showing its developmental trends. Finally, challenging issues, including the relevance of granular computing, in the said domain are elaborated as a future scope of research, together with some concerns. An extensive bibliography is also provided.
Improving Visual Reasoning Through Semantic Representation

In visual reasoning, the achievement of deep learning significantly improved the accuracy of results. Image features are primarily used as input to get answers. However, the image features are too redundant to learn accurate characterizations within a limited complexity and time. While in the process of human reasoning, abstract description of an image is usually to avoid irrelevant details. Inspired by this, a higher-level representation named semantic representation is introduced. In this paper, a detailed visual reasoning model is proposed. This new model contains an image understanding model based on semantic representation, feature extraction and process model refined with watershed and u-distance method, a feature vector learning model using pyramidal pooling and residual network, and a question understanding model combining problem embedding coding method and machine translation decoding method. The feature vector could better represent the whole image instead of overly focused on specific characteristics. The model using semantic representation as input verifies that more accurate results can be obtained by introducing a high-level semantic representation. The result also shows that it is feasible and effective to introduce high-level and abstract forms of knowledge representation into deep learning tasks. This study lays a theoretical and experimental foundation for introducing different levels of knowledge representation into deep learning in the future.
Long short term memory–convolutional neural network based deep hybrid approach for solar irradiance forecasting

The volatile behavior of solar energy is the biggest challenge in its successful integration with existing grid systems. Accurate global horizontal irradiance (GHI) forecasting can resolve this issue and lead to early and effective participation in the energy market. This study proposes a new hybrid deep learning model, namely long short term memory–convolutional neural network (LSTM–CNN), for hourly GHI forecasting, which models the spatio-temporal features by integrating the long short term memory (LSTM) and convolutional neural network (CNN) model. The proposed model is trained with the meteorological data of 23 locations of California State, USA, which includes temperature, precipitation, relative humidity, cloud cover, etc., as input parameters. The proposed hybrid LSTM–CNN model firstly uses LSTM to extract the temporal features from time-series solar irradiance data, followed by CNN, which extracts the spatial features from the correlation matrix of several meteorological variables of target and its neighbor location. The prediction accuracy of the developed model is analyzed rigorously by examining the performance for a year, for four seasons and under three sky conditions. Besides, the proposed LSTM–CNN model shows a forecast skill score in a range of about 37%–45% over few standalone models, including smart persistence, support vector machine, artificial neural network, LSTM, CNN and other hybrid models. The findings of the present work suggest that the proposed hybrid LSTM–CNN model is a reliable alternative for short-term GHI prediction due to its high predictive accuracy under diverse climatic, seasonal and sky conditions.
Day-ahead hourly photovoltaic power forecasting using attention-based CNN-LSTM neural network embedded with multiple relevant and target variables prediction pattern

Accurate forecasting of photovoltaic power plays a pivotal role in the integration, operation, and scheduling of smart grid systems. Notably, volatility and intermittence of solar energy are the primary constraints influencing the accuracy of photovoltaic power prediction. This work proposes, an attention-based long-term and short-term temporal neural network prediction model (ALSM) assembled using the convolutional neural network (CNN), long short-term memory neural network (LSTM), and attention mechanism under the multiple relevant and target variables prediction pattern (MRTPP). This is geared towards capturing the short-term and long-term temporal modes and achieving the day-ahead hourly photovoltaic power forecasting. The proposed method is verified by the historical data of the photovoltaic system downloaded from the DKASC website. Consequently, the results indicate that the forecasting accuracy using the MRTPP pattern is better than those common input-output prediction patterns. Moreover, the proposed ALSM model under the MRTPP pattern demonstrates more superiority compared to a few PV power forecasting methods including the statistical methods as well as artificial intelligence methods. Subsequently, different important parameters affecting the accuracy of forecasting range of the model are analyzed, and suggestions on memory lengths corresponding to the divergent prediction range are provided.
Distilling Knowledge via Knowledge Review

Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network. Previous methods mostly focus on proposing feature transformation and loss functions between the same level's features to improve the effectiveness. We differently study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our new review mechanism is effective and structurally simple. Our finally designed nested and compact framework requires negligible computation overhead, and outperforms other methods on a variety of tasks. We apply our method to classification, object detection, and instance segmentation tasks. All of them witness significant student network performance improvement.
Conflicting evidence combination from the perspective of networks

Dempster-Shafer evidence theory is widely used in the field of information fusion especially when confronting with uncertainties. However, Dempster’s rule of combination may lead to counter-intuitive results when dealing with highly conflicting bodies of evidence (BOEs). Numerous methods were proposed to address this problem. Enlightened by the research of interaction among nodes in complex networks, this paper study the combination of evidences from the perspective of networks: BOEs are regarded as nodes, the conflicting degree between BOEs is considered as one possible interaction between nodes. The direct and indirect interactions among nodes in networks are considered together to determine the weights of the BOEs. After process of weighted average, the modified BOEs can be efficiently combined by Dempster’s rule of combination. A numerical example is illustrated to show the use and better performance of the proposed method.
Pixel Difference Networks for Efficient Edge Detection

Recently, deep Convolutional Neural Networks (CNNs) can achieve human-level performance in edge detection with the rich and abstract edge representation capacities. However, the high performance of CNN based edge detection is achieved with a large pretrained CNN backbone, which is memory and energy consuming. In addition, it is surprising that the previous wisdom from the traditional edge detectors, such as Canny, Sobel, and LBP are rarely investigated in the rapid-developing deep learning era. To address these issues, we propose a simple, lightweight yet effective architecture named Pixel Difference Network (PiDiNet) for efficient edge detection. PiDiNet adopts novel pixel difference convolutions that integrate the traditional edge detection operators into the popular convolutional operations in modern CNNs for enhanced performance on the task, which enjoys the best of both worlds. Extensive experiments on BSDS500, NYUD, and Multicue are provided to demonstrate its effectiveness, and its high training and inference efficiency. Surprisingly, when training from scratch with only the BSDS500 and VOC datasets, PiDiNet can surpass the recorded result of human perception (0.807 vs. 0.803 in ODS F-measure) on the BSDS500 dataset with 100 FPS and less than 1M parameters. A faster version of PiDiNet with less than 0.1M parameters can still achieve comparable performance among state of the arts with 200 FPS. Results on the NYUD and Multicue datasets show similar observations. The codes are available at https://github.com/zhuoinoulu/pidinet.
Exploring Key Weather Factors From Analytical Modeling Toward Improved Solar Power Forecasting

Accurate solar power forecasting plays a critical role in ensuring the reliable and economic operation of power grids. Most of existing literature directly uses available weather conditions as input features, which might ignore some key weather factors and the coupling among weather conditions. Therefore, a novel solar power forecasting approach is proposed in this paper by exploring key weather factors from photovoltaic (PV) analytical modeling. The proposed approach is composed of three engines: (1) analytical modeling of PV systems; (2) machine learning methods for mapping weather features with solar power; and (3) a deviation analysis for solar power forecast adjustment. In contrast to the existing research that directly uses available weather conditions, this paper explores the physical knowledge from PV models. Different irradiance components and PV cell temperatures are derived from PV analytical modeling. These weather features are used to reformulate the input of machine learning methods, which helps achieve a better forecasting performance. Moreover, based on the historical forecasting deviations, a compensation term is presented to adjust the solar power forecast. Case studies based on measured datasets from PV systems in Australia demonstrate that the forecasting performance can be highly improved by taking advantage of the key weather features derived from PV models.
Deep Spatio-Temporal Representation for Detection of Road Accidents Using Stacked Autoencoder

Vision-based detection of road accidents using traffic surveillance video is a highly desirable but challenging task. In this paper, we propose a novel framework for automatic detection of road accidents in surveillance videos. The proposed framework automatically learns feature representation from the spatiotemporal volumes of raw pixel intensity instead of traditional hand-crafted features. We consider the accident of the vehicles as an unusual incident. The proposed framework extracts deep representation using denoising autoencoders trained over the normal traffic videos. The possibility of an accident is determined based on the reconstruction error and the likelihood of the deep representation. For the likelihood of the deep representation, an unsupervised model is trained using one class support vector machine. Also, the intersection points of the vehicle's trajectories are used to reduce the false alarm rate and increase the reliability of the overall system. We evaluated out proposed approach on real accident videos collected from the CCTV surveillance network of Hyderabad City in India. The experiments on these real accident videos demonstrate the efficacy of the proposed approach.
Network Layer Aspects of Permissionless Blockchains

Permissionless blockchains reach decentralized consensus without requiring pre-established identities or trusted third parties, thus enabling applications such as cryptocurrencies and smart contracts. Consensus is agreed on data that is generated by the application and transmitted by the system's (peer-to-peer) network layer. While many attacks on the network layer were discussed so far, there is no systematic approach that brings together known attacks, the requirements, and the design space of the network layer. In this paper, we survey attacks on the network layer of permissionless blockchains, and derive five requirements: 1) performance; 2) low cost of participation; 3) anonymity; 4) DoS resistance; and 5) topology hiding. Furthermore, we survey the design space of the network layer and qualitatively show the effect of each design decisions on the fulfillment of the requirements. Finally, we pick two aspects of the design space, in-band peer discovery and relay delay, and demonstrate possible directions of future research by quantitatively analyzing and optimizing simplified scenarios. We show that while most design decisions imply certain tradeoffs, there is a lack of models that analyze and formalize these tradeoffs. Such models could aid the design of the network layer of permissionless blockchains. One reason for the lack of models is the deliberately limited observability of deployed blockchains. We emphasize that simulation based approaches cope with these limitations and are suited for the analysis of the network layer of permissionless blockchains.
Neural Rendering and Reenactment of Human Actor Videos

We propose a method for generating video-realistic animations of real humans under user control. In contrast to conventional human character rendering, we do not require the availability of a production-quality photo-realistic three-dimensional (3D) model of the human but instead rely on a video sequence in conjunction with a (medium-quality) controllable 3D template model of the person. With that, our approach significantly reduces production cost compared to conventional rendering approaches based on production-quality 3D models and can also be used to realistically edit existing videos. Technically, this is achieved by training a neural network that translates simple synthetic images of a human character into realistic imagery. For training our networks, we first track the 3D motion of the person in the video using the template model and subsequently generate a synthetically rendered version of the video. These images are then used to train a conditional generative adversarial network that translates synthetic images of the 3D model into realistic imagery of the human. We evaluate our method for the reenactment of another person that is tracked to obtain the motion data, and show video results generated from artist-designed skeleton motion. Our results outperform the state of the art in learning-based human image synthesis.
To emoji or not to emoji? Examining the influence of emoji on consumer reactions to advertising

In recent years, advertisers have increasingly been using emoji in their promotional communications. However, little is known about how consumers might react to the use of emoji in advertisements. In this paper, we examine the effect of including emoji in advertisements on consumers' purchase intentions. Two empirical investigations (a laboratory experiment and an online study) provide convergent evidence that the presence of emoji leads consumers to experience higher positive affect, which in turn, leads to higher purchase intentions. We also examined the moderating influence of product type (utilitarian vs. hedonic) on the emoji ➔ positive affect ➔ purchase intentions link, finding that emoji are only effective for promoting hedonic products. The findings of our work offer theoretical and managerial implications with respect to why and when emoji are likely to make advertisements more effective.
Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training

We present an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition. Instead of explicitly combining multimodal information, which is commonplace in many state-of-the-art methods, we propose a different framework in which we embed the knowledge of multiple modalities in individual networks so that each unimodal network can achieve an improved performance. In particular, we dedicate separate networks per available modality and enforce them to collaborate and learn to develop networks with common semantics and better representations. We introduce a "spatiotemporal semantic alignment" loss (SSA) to align the content of the features from different networks. In addition, we regularize this loss with our proposed "focal regularization parameter" to avoid negative knowledge transfer. Experimental results show that our framework improves the test time recognition accuracy of unimodal networks, and provides the state-of-the-art performance on various dynamic hand gesture recognition datasets.
Fringe pattern denoising based on deep learning

In this paper, deep learning as a novel algorithm is proposed to reduce the noise of the fringe patterns. Usually, the training samples are acquired through experimental acquisition, but these data can be easily obtained by simulations in the proposed algorithm. Thus, the time cost used for the whole training process is greatly reduced. The performance of the proposed algorithm has been demonstrated through the analysis on the simulated and real fringe patterns. It is obvious that the proposed algorithm has a faster calculation speed compared with existing denoising algorithm, and recovers the fringe patterns with high quality. Most importantly, the proposed algorithm may provide a solution to other denoising problems in the field of optics, such as hologram and speckle denoising.
A provably secure certificateless public key encryption with keyword search

Public key encryption with keyword search, known as PEKS, is a useful technology in some Storage-as-a-Service clouds. This technology allows the cloud server to search on a keyword, which is received from the data owner/user, over encrypted data without having the ability to decrypt the stored data. PEKS with designated verifier, known as dPEKS, further advances the security of PEKS such that only a designated server can perform the keyword searching.Combining the certificateless technology, which aims to avoid the key escrow problem in ID-based cryptography, produces a certificateless dPEKS scheme that inherits the property of certificatelessness in key management in the dPEKS. This paper presents a provably secure certificateless dPEKS which has fixed an insecure scheme which was proposed earlier. The scheme presented in this paper is proven to be secure under Bilinear Diffie–Hellman problem.
Faster R-CNN for marine organisms detection and recognition using data augmentation

Recently, Faster Region-based Convolutional Neural Network (Faster R-CNN) has achieved marvelous accomplishment in object detection and recognition. In this paper, Faster R-CNN is applied to marine organisms detection and recognition. However, the training of Faster R-CNN requires a mass of labeled samples which are difficult to obtain for marine organisms. Therefore, three data augmentation methods dedicated to underwater-imaging are proposed. Specifically, the inverse process of underwater image restoration is used to simulate different marine turbulence environments. Perspective transformation is proposed to simulate different views of camera shooting. Illumination synthesis is used to simulate different marine uneven illuminating environments. The performance of each data augmentation method, together with previous frequently used data augmentation methods are evaluated by Faster R-CNN on the real-world underwater dataset, which validate the effectiveness of the proposed methods for marine organisms detection and recognition.
Hierarchical Convolutional Neural Networks for Segmentation of Breast Tumors in MRI With Application to Radiogenomics

Breast tumor segmentation based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a challenging problem and an active area of research. Particular challenges, similarly as in other segmentation problems, include the class-imbalance problem as well as confounding background in DCE-MR images. To address these issues, we propose a mask-guided hierarchical learning (MHL) framework for breast tumor segmentation via fully convolutional networks (FCN). Specifically, we first develop an FCN model to generate a 3D breast mask as the region of interest (ROI) for each image, to remove confounding information from input DCE-MR images. We then design a two-stage FCN model to perform coarse-to-fine segmentation for breast tumors. Particularly, we propose a Dice-Sensitivity-like loss function and a reinforcement sampling strategy to handle the class-imbalance problem. To precisely identify locations of tumors that underwent a biopsy, we further propose an FCN model to detect two landmarks located at two nipples. We finally selected the biopsied tumor based on both identified landmarks and segmentations. We validate our MHL method on 272 patients, achieving a mean Dice similarity coefficient (DSC) of 0.72 which is comparable to mutual DSC between expert radiologists. Using the segmented biopsied tumors, we also demonstrate that the automatically generated masks can be applied to radiogenomics and can identify luminal A subtype from other molecular subtypes with the similar accuracy with the analysis based on semi-manual tumor segmentation.
The effects of example-based explanations in a machine learning interface

The black-box nature of machine learning algorithms can make their predictions difficult to understand and explain to end-users. In this paper, we propose and evaluate two kinds of example-based explanations in the visual domain, normative explanations and comparative explanations (Figure 1), which automatically surface examples from the training set of a deep neural net sketch-recognition algorithm. To investigate their effects, we deployed these explanations to 1150 users on QuickDraw, an online platform where users draw images and see whether a recognizer has correctly guessed the intended drawing. When the algorithm failed to recognize the drawing, those who received normative explanations felt they had a better understanding of the system, and perceived the system to have higher capability. However, comparative explanations did not always improve perceptions of the algorithm, possibly because they sometimes exposed limitations of the algorithm and may have led to surprise. These findings suggest that examples can serve as a vehicle for explaining algorithmic behavior, but point to relative advantages and disadvantages of using different kinds of examples, depending on the goal.