Reviewerly/exHarmony huggingface dataset:
Deep Semantic-Alignment Hashing for Unsupervised Cross-Modal Retrieval

Deep hashing methods have achieved tremendous success in cross-modal retrieval, due to its low storage consumption and fast retrieval speed. In real cross-modal retrieval applications, it's hard to obtain label information. Recently, increasing attention has been paid to unsupervised cross-modal hashing. However, existing methods fail to exploit the intrinsic connections between images and their corresponding descriptions or tags (text modality). In this paper, we propose a novel Deep Semantic-Alignment Hashing (DSAH) for unsupervised cross-modal retrieval, which sufficiently utilizes the co-occurred image-text pairs. DSAH explores the similarity information of different modalities and we elaborately design a semantic-alignment loss function, which elegantly aligns the similarities between features with those between hash codes. Moreover, to further bridge the modality gap, we innovatively propose to reconstruct features of one modality with hash codes of the other one. Extensive experiments on three cross-modal retrieval datasets demonstrate that DSAH achieves the state-of-the-art performance.
Fundamental limits of quantum illumination

In quantum illumination (QI), a signal beam initially entangled with an idler beam held at the receiver interrogates a target region bathed in thermal background light. The returned beam is measured jointly with the idler in order to determine whether a weakly reflecting target is present. Using tools from quantum information theory, we derive lower bounds on the average error probability of detecting both specular and fading targets and on the mean squared error of estimating the reflectance of a detected target, which are obeyed by any QI transmitter satisfying a signal energy constraint. For bright thermal backgrounds, we show that the QI system using multiple copies of low-brightness two-mode squeezed vacuum states is nearly optimal. More generally, our results place limits on the best possible performance achievable using QI systems at all wavelengths, and at all signal and background noise levels.
STAViS: Spatio-Temporal AudioVisual Saliency Network

We introduce STAViS, a spatio-temporal audiovisual saliency network that combines spatio-temporal visual and auditory information in order to efficiently address the problem of saliency estimation in videos. Our approach employs a single network that combines visual saliency and auditory features and learns to appropriately localize sound sources and to fuse the two saliencies in order to obtain a final saliency map. The network has been designed, trained end-to-end, and evaluated on six different databases that contain audiovisual eye-tracking data of a large variety of videos. We compare our method against 8 different state-of-the-art visual saliency models. Evaluation results across databases indicate that our STAViS model outperforms our visual only variant as well as the other state-of-the-art models in the majority of cases. Also, the consistently good performance it achieves for all databases indicates that it is appropriate for estimating saliency "in-the-wild". The code is available at https://github.com/atsiami/STAViS.
Real-World Person Re-Identification via Degradation Invariance Learning

Person re-identification (Re-ID) in real-world scenarios usually suffers from various degradation factors, e.g., low-resolution, weak illumination, blurring and adverse weather. On the one hand, these degradations lead to severe discriminative information loss, which significantly obstructs identity representation learning; on the other hand, the feature mismatch problem caused by low-level visual variations greatly reduces retrieval performance. An intuitive solution to this problem is to utilize low-level image restoration methods to improve the image quality. However, existing restoration methods cannot directly serve to real-world Re-ID due to various limitations, e.g., the requirements of reference samples, domain gap between synthesis and reality, and incompatibility between low-level and high-level methods. In this paper, to solve the above problem, we propose a degradation invariance learning framework for real-world person Re-ID. By introducing a self-supervised disentangled representation learning strategy, our method is able to simultaneously extract identity-related robust features and remove real-world degradations without extra supervision. We use low-resolution images as the main demonstration, and experiments show that our approach is able to achieve state-of-the-art performance on several Re-ID benchmarks. In addition, our framework can be easily extended to other real-world degradation factors, such as weak illumination, with only a few modifications.
Semantic Pyramid for Image Generation

We present a novel GAN-based model that utilizes the space of deep features learned by a pre-trained classification model. Inspired by classical image pyramid representations, we construct our model as a Semantic Generation Pyramid -- a hierarchical framework which leverages the continuum of semantic information encapsulated in such deep features; this ranges from low level information contained in fine features to high level, semantic information contained in deeper features. More specifically, given a set of features extracted from a reference image, our model generates diverse image samples, each with matching features at each semantic level of the classification model. We demonstrate that our model results in a versatile and flexible framework that can be used in various classic and novel image generation tasks. These include: generating images with a controllable extent of semantic similarity to a reference image, and different manipulation tasks such as semantically-controlled inpainting and compositing; all achieved with the same model, with no further training.
Multi-Dimensional Pruning: A Unified Framework for Model Compression

In this work, we propose a unified model compression framework called Multi-Dimensional Pruning (MDP) to simultaneously compress the convolutional neural networks (CNNs) on multiple dimensions. In contrast to the existing model compression methods that only aim to reduce the redundancy along either the spatial/spatial-temporal dimension (e.g., spatial dimension for 2D CNNs, spatial and temporal dimensions for 3D CNNs) or the channel dimension, our newly proposed approach can simultaneously reduce the spatial/spatial-temporal and the channel redundancies for CNNs. Specifically, in order to reduce the redundancy along the spatial/spatial-temporal dimension, we downsample the input tensor of a convolutional layer, in which the scaling factor for the downsampling operation is adaptively selected by our approach. After the convolution operation, the output tensor is upsampled to the original size to ensure the unchanged input size for the subsequent CNN layers. To reduce the channel-wise redundancy, we introduce a gate for each channel of the output tensor as its importance score, in which the gate value is automatically learned. The channels with small importance scores will be removed after the model compression process. Our comprehensive experiments on four benchmark datasets demonstrate that our MDP framework outperforms the existing methods when pruning both 2D CNNs and 3D CNNs.
Temporal-Context Enhanced Detection of Heavily Occluded Pedestrians

State-of-the-art pedestrian detectors have performed promisingly on non-occluded pedestrians, yet they are still confronted by heavy occlusions. Although many previous works have attempted to alleviate the pedestrian occlusion issue, most of them rest on still images. In this paper, we exploit the local temporal context of pedestrians in videos and propose a tube feature aggregation network (TFAN) aiming at enhancing pedestrian detectors against severe occlusions. Specifically, for an occluded pedestrian in the current frame, we iteratively search for its relevant counterparts along temporal axis to form a tube. Then, features from the tube are aggregated according to an adaptive weight to enhance the feature representations of the occluded pedestrian. Furthermore, we devise a temporally discriminative embedding module (TDEM) and a part-based relation module (PRM), respectively, which adapts our approach to better handle tube drifting and heavy occlusions. Extensive experiments are conducted on three datasets, Caltech, NightOwls and KAIST, showing that our proposed method is significantly effective for heavily occluded pedestrian detection. Moreover, we achieve the state-of-the-art performance on the Caltech and NightOwls datasets.
Synthetic Learning: Learn From Distributed Asynchronized Discriminator GAN Without Sharing Medical Image Data

In this paper, we propose a data privacy-preserving and communication efficient distributed GAN learning framework named Distributed Asynchronized Discriminator GAN (AsynDGAN). Our proposed framework aims to train a central generator learns from distributed discriminator, and use the generated synthetic image solely to train the segmentation model. We validate the proposed framework on the application of health entities learning problem which is known to be privacy sensitive. Our experiments show that our approach: 1) could learn the real image’s distribution from multiple datasets without sharing the patient’s raw data. 2) is more efficient and requires lower bandwidth than other distributed deep learning methods. 3) achieves higher performance compared to the model trained by one real dataset, and almost the same performance compared to the model trained by all real datasets. 4) has provable guarantees that the generator could learn the distributed distribution in an all important fashion thus is unbiased.We release our AsynDGAN source code at: https://github.com/tommy-qichang/AsynDGAN
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation

We present MixNMatch, a conditional generative model that learns to disentangle and encode background, object pose, shape, and texture from real images with minimal supervision, for mix-and-match image generation. We build upon FineGAN, an unconditional generative model, to learn the desired disentanglement and image generator, and leverage adversarial joint image-code distribution matching to learn the latent factor encoders. MixNMatch requires bounding boxes during training to model background, but requires no other supervision. Through extensive experiments, we demonstrate MixNMatch's ability to accurately disentangle, encode, and combine multiple factors for mix-and-match image generation, including sketch2color, cartoon2img, and img2gif applications. Our code/models/demo can be found at https://github.com/Yuheng-Li/MixNMatch
OntoBestFit: A Best-Fit Occurrence Estimation strategy for RDF driven faceted semantic search

As Web 2.0 is transforming into a more systematized Semantic Web, there is an urgent need for semantic compliant techniques for web search. In this paper, the OntoBestFit has been proposed which is an RDF driven approach for minimizing ambiguity in search results and increasing the diversity of results, thereby solving both the context irrelevance and the serendipity problem. OntoBestFit focuses on harvesting RDF from standardized industry scale Semantic Wikis and transforming it into an intermediate dyadic structure, which makes it suitable for heterogeneous real-world domains. The approach focuses on deriving an RDF prioritization vector from a Term-Frequency Matrix and a Term co-occurrence Matrix formulated from the reduced dyadic RDF entities over a corpus of web pages to yield Query Indicator terms. The incorporation of dynamic query expansion by generating query facets from the Domain Ontologies and Query Indicator terms using the best-fit occurrence estimation algorithm has made this approach novel. The best-fit occurrence estimation algorithm is based on the adaptation of Simpson's Diversity Index to compute the reduced and highly appropriate semantically similar domain ontologies based on domain richness computation, to increase the diversity in search results. Also, an Enriched Adaptive Pointwise Mutual Information measure has been proposed to compute the semantic similarity. OntoBestFit yields an average accuracy of 94.91% with a very low False Discovery Rate of 0.07, with a response time of 0.28 ms which is the best in class semantic search strategy in the era of Semantic Web.
Bolt-Loosening Monitoring Framework Using an Image-Based Deep Learning and Graphical Model

In this study, we investigate a novel idea of using synthetic images of bolts which are generated from a graphical model to train a deep learning model for loosened bolt detection. Firstly, a framework for bolt-loosening detection using image-based deep learning and computer graphics is proposed. Next, the feasibility of the proposed framework is demonstrated through the bolt-loosening monitoring of a lab-scaled bolted joint model. For practicality, the proposed idea is evaluated on the real-scale bolted connections of a historical truss bridge in Danang, Vietnam. The results show that the deep learning model trained by the synthesized images can achieve accurate bolt recognitions and looseness detections. The proposed methodology could help to reduce the time and cost associated with the collection of high-quality training data and further accelerate the applicability of vision-based deep learning models trained on synthetic data in practice.
Polarized Reflection Removal With Perfect Alignment in the Wild

We present a novel formulation to removing reflection from polarized images in the wild. We first identify the misalignment issues of existing reflection removal datasets where the collected reflection-free images are not perfectly aligned with input mixed images due to glass refraction. Then we build a new dataset with more than 100 types of glass in which obtained transmission images are perfectly aligned with input mixed images. Second, capitalizing on the special relationship between reflection and polarized light, we propose a polarized reflection removal model with a two-stage architecture. In addition, we design a novel perceptual NCC loss that can improve the performance of reflection removal and general image decomposition tasks. We conduct extensive experiments, and results suggest that our model outperforms state-of-the-art methods on reflection removal.
Instance Shadow Detection

Instance shadow detection is a brand new problem, aiming to find shadow instances paired with object instances. To approach it, we first prepare a new dataset called SOBA, named after Shadow-OBject Association, with 3,623 pairs of shadow and object instances in 1,000 photos, each with individual labeled masks. Second, we design LISA, named after Light-guided Instance Shadow-object Association, an end-to-end framework to automatically predict the shadow and object instances, together with the shadow-object associations and light direction. Then, we pair up the predicted shadow and object instances, and match them with the predicted shadow-object associations to generate the final results. In our evaluations, we formulate a new metric named the shadow-object average precision to measure the performance of our results. Further, we conducted various experiments and demonstrate our method's applicability on light direction estimation and photo editing.
ScopeFlow: Dynamic Scene Scoping for Optical Flow

We propose to modify the common training protocols of optical flow, leading to sizable accuracy improvements without adding to the computational complexity of the training process. The improvement is based on observing the bias in sampling challenging data that exists in the current training protocol, and improving the sampling process. In addition, we find that both regularization and augmentation should decrease during the training protocol. Using an existing low parameters architecture, the method is ranked first on the MPI Sintel benchmark among all other methods, improving the best two frames method accuracy by more than 10%. The method also surpasses all similar architecture variants by more than 12% and 19.7% on the KITTI benchmarks, achieving the lowest Average End-Point Error on KITTI2012 among two-frame methods, without using extra datasets.
Genetic Programming With Image-Related Operators and a Flexible Program Structure for Feature Learning in Image Classification

Feature extraction is essential for solving image classification by transforming low-level pixel values into high-level features. However, extracting effective features from images is challenging due to high variations across images in scale, rotation, illumination, and background. Existing methods often have a fixed model complexity and require domain expertise. Genetic programming (GP) with a flexible representation can find the best solution without the use of domain knowledge. This article proposes a new GP-based approach to automatically learning informative features for different image classification tasks. In the new approach, a number of image-related operators, including filters, pooling operators, and feature extraction methods, are employed as functions. A flexible program structure is developed to integrate different functions and terminals into a single tree/solution. The new approach can evolve solutions of variable depths to extract various numbers and types of features from the images. The new approach is examined on 12 different image classification tasks of varying difficulty and compared with a large number of effective algorithms. The results show that the new approach achieves better classification performance than most benchmark methods. The analysis of the evolved programs/solutions and the visualization of the learned features provide deep insights on the proposed approach.
HEMOS: A novel deep learning-based fine-grained humor detecting method for sentiment analysis of social media

In this paper we introduce HEMOS (Humor-EMOji-Slang-based) system for fine-grained sentiment classification for the Chinese language using deep learning approach. We investigate the importance of recognizing the influence of humor, pictograms and slang on the task of affective processing of the social media. In the first step, we collected 576 frequent Internet slang expressions as a slang lexicon; then, we converted 109 Weibo emojis into textual features creating a Chinese emoji lexicon. In the next step, by performing two polarity annotations with new “optimistic humorous type” and “pessimistic humorous type” added to standard “positive” and “negative” sentiment categories, we applied both lexicons to attention-based bi-directional long short-term memory recurrent neural network (AttBiLSTM) and tested its performance on undersized labeled data. Our experimental results show that the proposed method can significantly improve the state-of-the-art methods in predicting sentiment polarity on Weibo, the largest Chinese social network.
Developing post-alert messaging for ShakeAlert, the earthquake early warning system for the West Coast of the United States of America

As ShakeAlert, the earthquake early warning system for the West Coast of the U.S., begins its transition to operational public alerting, we explore how post-alert messaging might represent system performance. Planned post-alert messaging can provide timely, crucial information to both emergency managers and ShakeAlert operators as well as calibrate expectations among various publics or public user groups and inform their responses to future alerts. There is a concern among the scientists and emergency managers that false alerts may negatively impact trust in the system, so quickly disseminated post-alert messages are necessary. For a new early warning system, such as ShakeAlert, this is particularly relevant given that the potentially affected population is likely to be unfamiliar with this system. We address this concern in six steps: (1) assessment of ShakeAlert performance to date, (2) characterization of human behavior and response to earthquake alerts, (3) presentation of a decision tree for issuing post-alert messages, (4) design of a critical set of post-alert messaging scenarios, (5) elaboration of these scenarios with message templates for a variety of communication channels, and (6) development of a typology of earthquake alerts. We further explore methods for monitoring and evaluating ShakeAlert post-alert messaging, for continuous improvement to the system.
Digital pathology and artificial intelligence will be key to supporting clinical and academic cellular pathology through COVID-19 and future crises: the PathLAKE consortium perspective

The measures to control the COVID-19 outbreak will likely remain a feature of our working lives until a suitable vaccine or treatment is found. The pandemic has had a substantial impact on clinical services, including cancer pathways. Pathologists are working remotely in many circumstances to protect themselves, colleagues, family members and the delivery of clinical services. The effects of COVID-19 on research and clinical trials have also been significant with changes to protocols, suspensions of studies and redeployment of resources to COVID-19. In this article, we explore the specific impact of COVID-19 on clinical and academic pathology and explore how digital pathology and artificial intelligence can play a key role to safeguarding clinical services and pathology-based research in the current climate and in the future.
A Conceptual Framework for Human–AI Hybrid Adaptivity in Education

Educational AI (AIEd) systems are increasingly designed and evaluated with an awareness of the hybrid nature of adaptivity in real-world educational settings. In practice, beyond being a property of AIEd systems alone, adaptivity is often jointly enacted by AI systems and human facilitators (e.g., teachers or peers). Despite much recent research activity, theoretical and conceptual guidance for the design of such human–AI systems remains limited. In this paper we explore how adaptivity may be shared across AIEd systems and the various human stakeholders who work with them. Based on a comparison of prior frameworks, which tend to examine adaptivity in AIEd systems or human coaches separately, we first synthesize a set of dimensions general enough to capture human–AI hybrid adaptivity. Using these dimensions, we then present a conceptual framework to map distinct ways in which humans and AIEd systems can augment each other’s abilities. Through examples, we illustrate how this framework can be used to characterize prior work and envision new possibilities for human–AI hybrid approaches in education.
Sentimental Analysis of Twitter Data with respect to General Elections in India

It is known that social media is one of the largest sources of unstructured data. Analyzing that data and harvesting meaning out of that is a tedious job. Recently opinion mining has become an emerging topic due to the vast amount of opinioned data available on the various social networking sites. Microblogging has appeared relatively recently, and twitter is the most popular microblogging sites used by the people. It is one of the biggest free, open data-source. In the world today, twitter often witnesses a lot of opinions. Opinion mining and sentiment analysis help researchers to gain insight into public emotions. In this paper, Twitter is used as a source of opinioned data. Twitter APIs are used for the collection of tweets. In this paper, R is used for the acquisition, pre- processing, analyzing the tweets, then sentiment analysis is performed based on the different approaches. In this paper, Tweets were collected from the period of Jan 2019 to March 2019. Using that tweets, sentiment analysis was performed to gain the opinion polarity of the folks concerning general elections held in India. Two candidates were considered for this study: Candidate-1 and Candidate-2. It was concluded that Candidate-1 is more liked and is famous as compared to Candidate-2. The result obtained in our paper was in full compliance with the actual election results obtained in May 2019.