Reviewerly/exHarmony huggingface dataset:
Towards Unique and Informative Captioning of Images

Despite considerable progress, state of the art image captioning models produce generic captions, leaving out important image details. Furthermore, these systems may even misrepresent the image in order to produce a simpler caption consisting of common concepts. In this paper, we first analyze both modern captioning systems and evaluation metrics through empirical experiments to quantify these phenomena. We find that modern captioning systems return higher likelihoods for incorrect distractor sentences compared to ground truth captions, and that evaluation metrics like SPICE can be ‘topped’ using simple captioning systems relying on object detectors. Inspired by these observations, we design a new metric (SPICE-U) by introducing a notion of uniqueness over the concepts generated in a caption. We show that SPICE-U is better correlated with human judgements compared to SPICE, and effectively captures notions of diversity and descriptiveness. Finally, we also demonstrate a general technique to improve any existing captioning model – by using mutual information as a re-ranking objective during decoding. Empirically, this results in more unique and informative captions, and improves three different state-of-the-art models on SPICE-U as well as average score over existing metrics (Code is available at https://github.com/princetonvisualai/SPICE-U ).
A hybrid and effective learning approach for Click Fraud detection

Click Fraud is a fraudulent act of clicking on pay-per-click advertisements to increase the site’s revenue or to drain revenue from the advertiser. This illegal act has been putting commercial industries in a dilemma for quite some time. These industries think twice before advertising their products on websites and mobile-apps, as many parties try to exploit them. To safely promote their products, there must be an efficient system to detect click fraud. To address this problem, we propose a model called CFXGB (Cascaded Forest and XGBoost). The proposed model, classified under supervised machine learning, is a combination of two learning models used for feature transformation and classification. We showcase its superior performance compared to other related models, and make a comparison with multiple click fraud datasets with varying sizes.
Feasible time-bin entanglement purification based on sum-frequency generation

High quality time-bin entanglement is widely exploited to achieve the purposes of fundamental tests of physics and the implementation of quantum communication protocols both in free space and optical fiber propagation. However, the imperfect approaches of generating time-bin entangled state will degrade its quality and limit its practical application. Entanglement purification is to distill high quality entangled states from low quality entangled states. In this paper, we present the first entanglement purification protocol (EPP) for time-bin entanglement. We first explain this EPP for two-photon time-bin entangled state and then extend it to the system of multi-photon time-bin entangled state. We also design a possible realization of this EPP with practical spontaneous parametric down conversion (SPDC) source. Differ from the conventional EPPs, this EPP does not require the sophisticated controlled-not (CNOT) gate or similar operations, and it uses the feasible sum-frequency generation (SFG) to perform the purification. Moreover, the double-pair noise emitted from the SPDC source can be eliminated automatically which is the other advantage of this EPP. If we combine with the faithful entanglement swapping, this EPP may have potential to be a part of full quantum repeaters.
A Survey of the State of Explainable AI for Natural Language Processing.

Recent years have seen important advances in the quality of state-of-the-art models, but this has come at the expense of models becoming less interpretable. This survey presents an overview of the current state of Explainable AI (XAI), considered within the domain of Natural Language Processing (NLP). We discuss the main categorization of explanations, as well as the various ways explanations can be arrived at and visualized. We detail the operations and explainability techniques currently available for generating explanations for NLP model predictions, to serve as a resource for model developers in the community. Finally, we point out the current gaps and encourage directions for future work in this important research area.
Dynamic Federated Learning for GMEC With Time-Varying Wireless Link

Smart grid applications, such as predicting energy consumption, grid user behavior analysis and predicting energy theft, etc., are data-driven applications that require machine learning with a wealth of data generated from Internet of Things (IoT) based metering devices. However, traditional methods of uploading this huge data to the remote cloud for data analytics may be low efficient due to the non-negligible network transmission delay. By deploying a number of computing-enabled devices at the network edge, edge computing supports the implementation of machine learning close to the power grid environment. Considering the limited computing resources of edge devices and non-independent and identical (non-IID) data source, federated learning is a feasible edge computing based machine learning model. In federated learning, distributed mobile clients and a federated server collaborate to perform machine learning. Generally, the more clients to join the federated learning, the faster to obtain learning convergence and the higher resource utility. However, the communications between clients and the server in training rounds of federated learning may fail due to time-varying link reliability properties in a wireless network of smart grid, which not only slows down the model convergence rate but also wastes resources, such as energy consumption for invalid local training. This paper studies a dynamic federated learning problem in a power grid mobile edge computing (GMEC) environment, considering the high dynamic of link reliability. We design a delay deadline constrained federated learning framework to avoid extremely long training delay, and then formulate a dynamic client selection problem for computing utility maximization in such learning framework. Two online client selection algorithms, including cli-max greedy and uti-positive guarantee, are proposed to address the problem. The theoretical analysis and simulation results are conducted to illustrate the efficiency of the proposal.
AI Ethics: A Long History and a Recent Burst of Attention

Artificial intelligence (AI) ethics has become a hot topic in the popular press and in scholarly writing. In this column, five noted scholars give their opinions on what AI issues will become important in the foreseeable future.
On Analyzing COVID-19-related Hate Speech Using BERT Attention

The emergence of COVID-19 has engendered a new wave of online hate speech in social media platforms such as Twitter.Its widespread effects range from acts of cyberharassment towards certain ethnic communities (e.g., the Asian community), to targeting older people belonging to age groups correlated with higher mortality rates (termed infamously as "Boomer Remover").Thus, an urgent need arises for a timely mitigation of this new wave of online hate speech.In this work, we aim to discover the hate-related keywords linked to COVID-19 in hateful tweets posted on Twitter so that users posting such keywords can be asked to reconsider posting them.We first collect a new dataset of tweets targeting older people supplementing with a dataset targeting the Asian community.Then, we develop an approach to analyze the datasets with BERT (a transformer-based model) attention mechanism and discover 186 novel keywords targeting the Asian community and 100 keywords targeting older people.Based on our study, we then propose a control mechanism wherein a user can be asked to reconsider using certain sensitive words identified by our approach.We further perform an exploratory analysis of BERT attention mechanism and find that the most high-impact, long distance attentions are learned in the earlier or later layers of the model depending on the underlying data distribution.Our study indicates that the BERT model in some cases uses a hate keyword and an associated group or individual to make predictions, a finding that is inline with existing hate-speech research, which suggests that hate-speech is often aimed at certain groups or individuals.
Efficient steganalysis using convolutional auto encoder network to ensure original image quality

Steganalysis is the process of analyzing and predicting the presence of hidden information in images. Steganalysis would be most useful to predict whether the received images contain useful information. However, it is more difficult to predict the hidden information in images which is computationally difficult. In the existing research method, this is resolved by introducing the deep learning approach which attempts to perform steganalysis tasks in effectively. However, this research method does not concentrate the noises present in the images. It might increase the computational overhead where the error cost adjustment would require more iteration. This is resolved in the proposed research technique by introducing the novel research method called Non-Gaussian Noise Aware Auto Encoder Convolutional Neural Network (NGN-AEDNN). Classification technique provides a more flexible way for steganalysis where the multiple features present in the environment would lead to an inaccurate prediction rate. Here, learning accuracy is improved by introducing noise removal techniques before performing a learning task. Non-Gaussian Noise Removal technique is utilized to remove the noises before learning. Also, Gaussian noise removal is applied at every iteration of the neural network to adjust the error rate without the involvement of noisy features. This proposed work can ensure efficient steganalysis by accurate learning task. Matlab has been employed to implement the method by performing simulations from which it is proved that the proposed research technique NGN-AEDNN can ensure the efficient steganalysis outcome with the reduced computational overhead when compared with the existing methods.
Passive continuous-variable quantum key distribution using a locally generated local oscillator

We proposed a passive continuous-variable quantum key distribution (CVQKD) using a locally generated local oscillator (LLO) at reception. Different from the LLO-based Gaussian-modulated coherent-state quantum key distribution, our protocol is implemented by taking advantage of a multimode thermal source, beam splitters, optical attenuators, and homodyne detectors rather than the amplitude and phase modulators, which could be a cost-effective solution for LLO-based CVQKD. We adopt a general LLO noise model to analyze the performance of our protocol. That is to say, our consideration for noise in the proposed scheme contains the excess noise caused by passive state preparation, the phase noise, and the photon-leakage noise from the reference path. Because our protocol waives the necessity of the use of amplitude and phase modulators, there is no need to bear the effect of modulation noise caused by finite dynamics. Simulation results show that the improvement of the mode-overlap coefficient can suppress the excess noise caused by passive state preparation and thus enhance the performance of our protocol. Because lower relative phase drift and the photon-leakage noise can be obtained by using the multiplexing technique, we can still achieve reasonable performance of our protocol with the mode-overlap coefficient values which are very close to the experimental reference. Furthermore, we perform the finite-size analysis for the proposed protocol, which is more practical than that achieved in the asymptotic limit. This work confirms the feasibility of passive CVQKD implemented in the LLO configuration within the metropolitan area.
Load Balancing Techniques in Cloud Computing: Extensive Review

It has become difficult to handle traditional networks because of extensive network developments and an increase in the number of network users, and also because of new technologies like cloud computing and big data.Traditional networks are experiencing an increase in VM load and in the time taken for processing tasks.Hence, it has become essential to modify the traditional network architecture.A notion called Load balancing techniques that increases the conformance of network management was presented recently to deal with this problem.The critical need for load balancing emerges due to network resources limitations and requirements fulfillment that facilitates traffic distribution through various resources to enhance the efficiency and reliability of network resources.This task has been carried out by several researchers before, who have presented various algorithms with their benefits and shortcomings.The focus of this research is on the notion of cloud computing load balancing and on the advantages and disadvantages of a chosen load balancing algorithm.Furthermore, it examines the metrics and issues of these algorithms.
Spotting Football Events Using Two-Stream Convolutional Neural Network and Dilated Recurrent Neural Network

This paper addresses the problem of event detection and localization in long football (soccer) videos. Our key idea is that understanding the long-range dependencies between video frames is imperative for accurate event localization in long football videos. Additionally, proper event detection is not likely for fast movements in football videos without considering mid-range and short-range correlations between neighboring video frames. We argue that event spotting can be considerably improved by considering short-range to long-range frame dependencies in a unified architecture. To model long-range and mid-range dependencies, we propose to use the dilated recurrent neural network (DilatedRNN) with long short-term memory (LSTM) units, grounded on two-stream convolutional neural network (Two-stream CNN) features. While two-stream CNN extracts local spatiotemporal features necessary for fine-level details, the DilatedRNN makes the information obtained from distant frames available for the classifier and spotting algorithms. Evaluating our event spotting algorithm on the largest publicly available benchmark football dataset –SoccerNet– shows an accuracy improvement of 0.8% - 13.6% compared to state of the art, and up to 30.1% accuracy gain in comparison to the baselines. We also investigate the contribution of each neural network component in spotting accuracy through an extensive ablation study.
Multimodal Deep Learning Framework for Image Popularity Prediction on Social Media

Billions of photos are uploaded to the Web daily through various types of social networks. Some of these images receive millions of views and become popular, whereas others remain completely unnoticed. This raises the problem of predicting image popularity on social media. The popularity of an image can be affected by several factors, such as visual content, aesthetic quality, user, post metadata, and time. Thus, considering all these factors is essential for accurately predicting image popularity. In addition, the efficiency of the predictive model also plays a crucial role. In this study, motivated by multimodal learning, which uses information from various modalities, and the current success of convolutional neural networks (CNNs) in various fields, we propose a deep learning model, called visual-social CNN (VSCNN), which predicts the popularity of a posted image by incorporating various types of visual and social features into a unified network model. VSCNN first learns to extract high-level representations from the input visual and social features by utilizing two individual CNNs. The outputs of these two networks are then fused into a joint network to estimate the popularity score in the output layer. We assess the performance of the proposed method by conducting extensive experiments on a data set of approximately 432K images posted on Flickr. The simulation results demonstrate that the proposed VSCNN model significantly outperforms state-of-the-art models, with a relative improvement of greater than 2.33%, 7.59%, and 14.16% in terms of Spearman's Rho, mean absolute error, and mean-squared error, respectively.
Hybrid harmony search algorithm for social network contact tracing of COVID-19

The coronavirus disease 2019 (COVID-19) was first reported in December 2019 in Wuhan, China, and then moved to almost every country showing an unprecedented outbreak. The world health organization declared COVID-19 a pandemic. Since then, millions of people were infected, and millions have lost their lives all around the globe. By the end of 2020, effective vaccines that could prevent the fast spread of the disease started to loom on the horizon. Nevertheless, isolation, social distancing, face masks, and quarantine are the best-known measures, in the time being, to fight the pandemic. On the other hand, contact tracing is an effective procedure in tracking infections and saving others' lives. In this paper, we devise a new approach using a hybrid harmony search (HHS) algorithm that casts the problem of finding strongly connected components (SCCs) to contact tracing. This new approach is named as hybrid harmony search contact tracing (HHS-CT) algorithm. The hybridization is achieved by integrating the stochastic hill climbing into the operators' design of the harmony search algorithm. The HHS-CT algorithm is compared to other existing algorithms of finding SCCs in directed graphs, where it showed its superiority over these algorithms. The devised approach provides a 77.18% enhancement in terms of run time and an exceptional average error rate of 1.7% compared to the other existing algorithms of finding SCCs.
General virtual sketching framework for vector line art

Vector line art plays an important role in graphic design, however, it is tedious to manually create. We introduce a general framework to produce line drawings from a wide variety of images, by learning a mapping from raster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable rasterization module allows for training with only supervised raster data. We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules. Furthermore, we develop a stroke regularization loss that encourages the model to use fewer and longer strokes to simplify the resulting vector image. Ablation studies and comparisons with existing methods corroborate the efficiency of our approach which is able to generate visually better results in less computation time, while generalizing better to a diversity of images and applications.
Robustness Verification of Semantic Segmentation Neural Networks Using Relaxed Reachability

Abstract This paper introduces robustness verification for semantic segmentation neural networks (in short, semantic segmentation networks [SSNs]), building on and extending recent approaches for robustness verification of image classification neural networks. Despite recent progress in developing verification methods for specifications such as local adversarial robustness in deep neural networks (DNNs) in terms of scalability, precision, and applicability to different network architectures, layers, and activation functions, robustness verification of semantic segmentation has not yet been considered. We address this limitation by developing and applying new robustness analysis methods for several segmentation neural network architectures, specifically by addressing reachability analysis of up-sampling layers, such as transposed convolution and dilated convolution. We consider several definitions of robustness for segmentation, such as the percentage of pixels in the output that can be proven robust under different adversarial perturbations, and a robust variant of intersection-over-union (IoU), the typical performance evaluation measure for segmentation tasks. Our approach is based on a new relaxed reachability method, allowing users to select the percentage of a number of linear programming problems (LPs) to solve when constructing the reachable set, through a relaxation factor percentage. The approach is implemented within NNV, then applied and evaluated on segmentation datasets, such as a multi-digit variant of MNIST known as M2NIST. Thorough experiments show that by using transposed convolution for up-sampling and average-pooling for down-sampling, combined with minimizing the number of ReLU layers in the SSNs, we can obtain SSNs with not only high accuracy (IoU), but also that are more robust to adversarial attacks and amenable to verification. Additionally, using our new relaxed reachability method, we can significantly reduce the verification time for neural networks whose ReLU layers dominate the total analysis time, even in classification tasks.
Pattern-enhanced Contrastive Policy Learning Network for Sequential Recommendation

Sequential recommendation aims to predict users’ future behaviors given their historical interactions. However, due to the randomness and diversity of a user’s behaviors, not all historical items are informative to tell his/her next choice. It is obvious that identifying relevant items and extracting meaningful sequential patterns are necessary for a better recommendation. Unfortunately, few works have focused on this sequence denoising process. In this paper, we propose a PatteRn-enhanced ContrAstive Policy Learning Network (RAP for short) for sequential recommendation, RAP formalizes the denoising problem in the form of Markov Decision Process (MDP), and sample actions for each item to determine whether it is relevant with the target item. To tackle the lack of relevance supervision, RAP fuses a series of mined sequential patterns into the policy learning process, which work as a prior knowledge to guide the denoising process. After that, RAP splits the initial item sequence into two disjoint subsequences: a positive subsequence and a negative subsequence. At this, a novel contrastive learning mechanism is introduced to guide the sequence denoising and achieve preference estimation from the positive subsequence simultaneously. Extensive experiments on four public real-world datasets demonstrate the effectiveness of our approach for sequential recommendation.