Reviewerly/exHarmony huggingface dataset:
Deep Generative Adversarial Residual Convolutional Networks for Real-World Super-Resolution

Most current deep learning based single image super-resolution (SISR) methods focus on designing deeper / wider models to learn the non-linear mapping between low-resolution (LR) inputs and the high-resolution (HR) outputs from a large number of paired (LR/HR) training data. They usually take as assumption that the LR image is a bicubic down-sampled version of the HR image. However, such degradation process is not available in real-world settings i.e. inherent sensor noise, stochastic noise, compression artifacts, possible mismatch between image degradation process and camera device. It reduces significantly the performance of current SISR methods due to real-world image corruptions. To address these problems, we propose a deep Super-Resolution Residual Convolutional Generative Adversarial Network (SRResCGAN <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> ) to follow the real-world degradation settings by adversarial training the model with pixel-wise supervision in the HR domain from its generated LR counterpart. The proposed network exploits the residual learning by minimizing the energy-based objective function with powerful image regularization and convex optimization techniques. We demonstrate our proposed approach in quantitative and qualitative experiments that generalize robustly to real input and it is easy to deploy for other downscaling operators and mobile/embedded devices.
Deep Learning-Based Wrapped Phase Denoising Method for Application in Digital Holographic Speckle Pattern Interferometry

This paper presents a new processing method for denoising interferograms obtained by digital holographic speckle pattern interferometry (DHSPI) to serve in the structural diagnosis of artworks. DHSPI is a non-destructive and non-contact imaging method that has been successfully applied to the structural diagnosis of artworks by detecting hidden subsurface defects and quantifying the deformation directly from the surface illuminated by coherent light. The spatial information of structural defects is mostly delivered as local distortions interrupting the smooth distribution of intensity during the phase-shifted formation of fringe patterns. Distortions in fringe patterns are recorded and observed from the estimated wrapped phase map, but the inevitable electronic speckle noise directly affects the quality of the image and consequently the assessment of defects. An effective method for denoising DHSPI wrapped phase based on deep learning is presented in this paper. Although a related method applied to interferometry for reducing Gaussian noise has been introduced, it is not suitable for application in DHSPI to reduce speckle noise. Thus, the paper proposes a new method to remove speckle noise in the wrapped phase. Simulated data and experimental captured data from samples prove that the proposed method can effectively reduce the speckle noise of the DHSPI wrapped phase to extract the desired information. The proposed method is helpful for accurately detecting defects in complex defect topography maps and may help to accelerate defect detection and characterization procedures.
VPLNet: Deep Single View Normal Estimation With Vanishing Points and Lines

We present a novel single-view surface normal estimation method that combines traditional line and vanishing point analysis with a deep learning approach. Starting from a color image and a Manhattan line map, we use a deep neural network to regress on a dense normal map, and a dense Manhattan label map that identifies planar regions aligned with the Manhattan directions. We fuse the normal map and label map in a fully differentiable manner to produce a refined normal map as final output. To do so, we softly decompose the output into a Manhattan part and a non-Manhattan part. The Manhattan part is treated by discrete classification and vanishing points, while the non-Manhattan part is learned by direct supervision. Our method achieves state-of-the-art results on standard single-view normal estimation benchmarks. More importantly, we show that by using vanishing points and lines, our method has better generalization ability than existing works. In addition, we demonstrate how our surface normal network can improve the performance of depth estimation networks, both quantitatively and qualitatively, in particular, in 3D reconstructions of walls and other flat surfaces.
Memetic algorithms outperform evolutionary algorithms in multimodal optimisation

Memetic algorithms integrate local search into an evolutionary algorithm to combine the advantages of rapid exploitation and global optimisation. We provide a rigorous runtime analysis of memetic algorithms on the Hurdle problem, a landscape class of tunable difficulty with a "big valley structure", a characteristic feature of many hard combinatorial optimisation problems. A parameter called hurdle width describes the length of fitness valleys that need to be overcome. We show that the expected runtime of plain evolutionary algorithms like the (1+1) EA increases steeply with the hurdle width, yielding superpolynomial times to find the optimum, whereas a simple memetic algorithm, (1+1) MA, only needs polynomial expected time. Surprisingly, while increasing the hurdle width makes the problem harder for evolutionary algorithms, it becomes easier for memetic algorithms. We further give the first rigorous proof that crossover can decrease the expected runtime in memetic algorithms. A (2+1) MA using mutation, crossover and local search outperforms any other combination of these operators. Our results demonstrate the power of memetic algorithms for problems with big valley structures and the benefits of hybridising multiple search operators.
BiANE

Network embedding effectively transforms complex network data into a low-dimensional vector space and has shown great performance in many real-world scenarios, such as link prediction, node classification, and similarity search. A plethora of methods have been proposed to learn node representations and achieve encouraging results. Nevertheless, little attention has been paid on the embedding technique for bipartite attributed networks, which is a typical data structure for modeling nodes from two distinct partitions.
“Looking at the Right Stuff” – Guided Semantic-Gaze for Autonomous Driving

In recent years, predicting driver's focus of attention has been a very active area of research in the autonomous driving community. Unfortunately, existing state-of-the-art techniques achieve this by relying only on human gaze information, thereby ignoring scene semantics. We propose a novel Semantics Augmented GazE (SAGE) detection approach that captures driving specific contextual information, in addition to the raw gaze. Such a combined attention mechanism serves as a powerful tool to focus on the relevant regions in an image frame in order to make driving both safe and efficient. Using this, we design a complete saliency prediction framework - SAGE-Net, which modifies the initial prediction from SAGE by taking into account vital aspects such as distance to objects (depth), ego vehicle speed, and pedestrian crossing intent. Exhaustive experiments conducted through four popular saliency algorithms show that on 49/56 (87.5%) cases - considering both the overall dataset and crucial driving scenarios, SAGE outperforms existing techniques without any additional computational overhead during the training process. The augmented dataset along with the relevant code are available as part of the supplementary material.
Deep Reinforcement Polishing Network for Video Captioning

The video captioning task aims to describe video content using several natural-language sentences. Although one-step encoder-decoder models have achieved promising progress, the generations always involve many errors, which are mainly caused by the large semantic gap between the visual domain and the language domain and by the difficulty in long-sequence generation. The underlying challenge of video captioning, i.e., sequence-to-sequence mapping across different domains, is still not well handled. Inspired by the proofreading procedure of human beings, the generated caption can be gradually polished to improve its quality. In this paper, we propose a deep reinforcement polishing network (DRPN) to refine the caption candidates, which consists of a word-denoising network (WDN) to revise word errors and a grammar-checking network (GCN) to revise grammar errors. On the one hand, the long-term reward in deep reinforcement learning benefits the long-sequence generation, which takes the global quality of caption sentences into account. On the other hand, the caption candidate can be considered a bridge between visual and language domains, where the semantic gap is gradually reduced with better candidates generated by repeated revisions. In experiments, we present adequate evaluations to show that the proposed DRPN achieves comparable and even better performance than the state-of-the-art methods. Furthermore, the DRPN is model-irrelevant and can be integrated into any video captioning models to refine their generated caption sentences.
NETNet: Neighbor Erasing and Transferring Network for Better Single Shot Object Detection

Due to the advantages of real-time detection and improved performance, single-shot detectors have gained great attention recently. To solve the complex scale variations, single-shot detectors make scale-aware predictions based on multiple pyramid layers. However, the features in the pyramid are not scale-aware enough, which limits the detection performance. Two common problems in single-shot detectors caused by object scale variations can be observed: (1) small objects are easily missed; (2) the salient part of a large object is sometimes detected as an object. With this observation, we propose a new Neighbor Erasing and Transferring (NET) mechanism to reconfigure the pyramid features and explore scale-aware features. In NET, a Neighbor Erasing Module (NEM) is designed to erase the salient features of large objects and emphasize the features of small objects in shallow layers. A Neighbor Transferring Module (NTM) is introduced to transfer the erased features and highlight large objects in deep layers. With this mechanism, a single-shot network called NETNet is constructed for scale-aware object detection. In addition, we propose to aggregate nearest neighboring pyramid features to enhance our NET. NETNet achieves 38.5% AP at a speed of 27 FPS and 32.0% AP at a speed of 55 FPS on MS COCO dataset. As a result, NETNet achieves a better trade-off for real-time and accurate object detection.
Decoding design agendas

If we expect our children to be driving technology design agendas in the future, we must first help them recognize that opinions and beliefs are baked into the technologies that we create and that these opinions may serve some groups of people more than others. In this paper, we discuss an ethical design activity completed by 19 middle school-aged children. The activity encourages students to see technical systems as socio-technical systems, to engage them in stakeholder analysis, and to apply ethical design tools in order to redesign YouTube. Results indicate students are capable of transforming into critical users and ethical designers of technology. They are able to recognize the underlying design agendas for popular technologies such as YouTube, identify stakeholders who shape those design agendas, and apply an array of tools to reimagine technologies in a more inclusive manner.
Multiauthority Access Control With Anonymous Authentication for Personal Health Record

A personal health record (PHR) system is a smart health system that serves patients and doctors. A PHR is usually stored in a cloud and managed by a semitrusted cloud provider. However, there is still a possibility of the exposure of personal health information to semitrusted parties and unauthorized users. To protect the privacy of patients and ensure that patients can control their PHRs, a patient-centric PHR sharing framework is proposed in this article. In this framework, all PHRs are protected with multiauthority attribute-based encryption before outsourcing, which solves the key hosting problem and achieves fine-grained access control to PHRs. Furthermore, an anonymous authentication between the cloud and the user is proposed to ensure data integrity on the cloud while not exposing the user's identity during authentication. The proposed authentication is issued from a new online-offline attribute-based signature. It can make the encrypted PHRs resist collusion attacks and not be forged during the period of sharing, which enhances patients' control of their PHRs. Online-offline and outsourcing decryption also reduces calculation costs and improves operational efficiency. Finally, comparisons are given based on numerical experiments.
A Kernel Density Estimation Based Loss Function and its Application to ASV-Spoofing Detection

Biometric systems are exposed to spoofing attacks which may compromise their security, and voice biometrics, also known as automatic speaker verification (ASV), is no exception. Replay, synthesis and voice conversion attacks cause false acceptances that can be detected by anti-spoofing systems. Recently, deep neural networks (DNNs) which extract embedding vectors have shown superior performance than conventional systems in both ASV and anti-spoofing tasks. In this work, we develop a new concept of loss function for training DNNs which is based on kernel density estimation (KDE) techniques. The proposed loss functions estimate the probability density function (pdf) of every training class in each mini-batch, and compute a log likelihood matrix between the embedding vectors and pdfs of all training classes within the mini-batch in order to obtain the KDE-based loss. To evaluate our proposal for spoofing detection, experiments were carried out on the recent ASVspoof 2019 corpus, including both logical and physical access scenarios. The experimental results show that training a DNN based anti-spoofing system with our proposed loss functions clearly outperforms the performance of the same system being trained with other well-known loss functions. Moreover, the results also show that the proposed loss functions are effective for different types of neural network architectures.
A multispectral feature fusion network for robust pedestrian detection

The multispectral information, including both visible information and infrared information, can describe the detection target in a comprehensive manner. The deep learning (DL)-based detectors that fuse the multispectral features can detect pedestrians robustly in various environments. Therefore, this paper puts forward a robust multispectral feature fusion network (MSFFN) for pedestrian detection, which fully integrates the features extracted from visible light and infrared channels. Specifically, multiscale semantic features were extracted by two core modules, namely, multiscale feature extraction of visible images (MFEV) and multiscale feature extraction of infrared images (MFEI), and fused by the improved YOLOv3 network for pedestrian recognition. Through experiments on the KAIST dataset, it is proved that the MSFFN model can detect pedestrians more accurately than both MFEV and MFEI over daytime and nighttime images on multiple scales. The experimental results on the KAIST multispectral data set in the last section showed that our proposed MFMFN model was superior to a number of state-of-the-art multispectral pedestrian detectors methods in accuracy and speed. The model was also found to strike a good balance between accuracy and speed, and perform excellently on small input images. The research results shed important new light on the design of self-driving vehicles.
Deep Shutter Unrolling Network

We present a novel network for rolling shutter effect correction. Our network takes two consecutive rolling shutter images and estimates the corresponding global shutter image of the latest frame. The dense displacement field from a rolling shutter image to its corresponding global shutter image is estimated via a motion estimation network. The learned feature representation of a rolling shutter image is then warped, via the displacement field, to its global shutter representation by a differentiable forward warping block. An image decoder recovers the global shutter image based on the warped feature representation. Our network can be trained end-to-end and only requires the global shutter image for supervision. Since there is no public dataset available, we also propose two large datasets: the Carla-RS dataset and the Fastec-RS dataset. Experimental results demonstrate that our network outperforms the state-of-the-art methods. We make both our code and datasets available at https://github.com/ethliup/DeepUnrollNet.
An empirical investigation of performance overhead in cross-platform mobile development frameworks

Abstract The heterogeneity of the leading mobile platforms in terms of user interfaces, user experience, programming language, and ecosystem have made cross-platform development frameworks popular. These aid the creation of mobile applications – apps – that can be executed across the target platforms (typically Android and iOS) with minimal to no platform-specific code. Due to the cost- and time-saving possibilities introduced through adopting such a framework, researchers and practitioners alike have taken an interest in the underlying technologies. Examining the body of knowledge, we, nonetheless, frequently encounter discussions on the drawbacks of these frameworks, especially with regard to the performance of the apps they generate. Motivated by the ongoing discourse and a lack of empirical evidence, we scrutinised the essential piece of the cross-platform frameworks: the bridge enabling cross-platform code to communicate with the underlying operating system and device hardware APIs. The study we present in the article benchmarks and measures the performance of this bridge to reveal its associated overhead in Android apps. The development of the artifacts for this experiment was conducted using five cross-platform development frameworks to generate Android apps, in addition to a baseline native Android app implementation. Our results indicate that – for Android apps – the use of cross-platform frameworks for the development of mobile apps may lead to decreased performance compared to the native development approach. Nevertheless, certain cross-platform frameworks can perform equally well or even better than native on certain metrics which highlights the importance of well-defined technical requirements and specifications for deliberate selection of a cross-platform framework or overall development approach.
Multi-Task Travel Route Planning With a Flexible Deep Learning Framework

Travel route planning aims to map out a feasible sightseeing itinerary for a traveler covering famous attractions and meeting the tourist's desire. It is very useful for tourists to plan their travel routes when they want to travel at unfamiliar scenic cities. Existing methods for travel route planning mainly concentrate on a single planning problem for a special task, but is not capable of being applied to other tasks. For example, previous must-visit planning methods cannot be applied to the next-point recommendation, despite these two tasks are closely related to each other in travel route planning. Besides, most of the existing work do not consider the important auxiliary information such as Point of Interests (POI) attributes, user preference, and historical route data in their approaches. In this paper, we propose a flexible Multi-task Deep Travel Route Planning framework named MDTRP to integrate rich auxiliary information for more effective planning. Specifically, we first construct a heterogeneous network through the relations between users and POIs and employ a heterogeneous network embedding method to learn the features of users and POIs. Then we present an attention-based deep model to integrate the auxiliary information and focus on important visited points for the prediction of next POIs. Finally, a beam search algorithm is introduced to flexibly generate multiple feasible route candidates for three types of planning tasks (next-point recommendation, general route planning, and must-visit planning). We introduce six public datasets to conduct extensive experiments, of which the results demonstrate the flexibility and superiority of the proposed approach in travel route planning.
Visual-Textual Capsule Routing for Text-Based Video Segmentation

Joint understanding of vision and natural language is a challenging problem with a wide range of applications in artificial intelligence. In this work, we focus on integration of video and text for the task of actor and action video segmentation from a sentence. We propose a capsule-based approach which performs pixel-level localization based on a natural language query describing the actor of interest. We encode both the video and textual input in the form of capsules, which provide a more effective representation in comparison with standard convolution based features. Our novel visual-textual routing mechanism allows for the fusion of video and text capsules to successfully localize the actor and action. The existing works on actor-action localization are mainly focused on localization in a single frame instead of the full video. Different from existing works, we propose to perform the localization on all frames of the video. To validate the potential of the proposed network for actor and action video localization, we extend an existing actor-action dataset (A2D) with annotations for all the frames. The experimental evaluation demonstrates the effectiveness of our capsule network for text selective actor and action localization in videos. The proposed method also improves upon the performance of the existing state-of-the art works on single frame-based localization.
Robust Layout-aware IE for Visually Rich Documents with Pre-trained Language Models

Many business documents processed in modern NLP and IR pipelines are visually rich: in addition to text, their semantics can also be captured by visual traits such as layout, format, and fonts. We study the problem of information extraction from visually rich documents (VRDs) and present a model that combines the power of large pre-trained language models and graph neural networks to efficiently encode both textual and visual information in business documents. We further introduce new fine-tuning objectives to improve in-domain unsupervised fine-tuning to better utilize large amount of unlabeled in-domain data.