Reviewerly/exHarmony huggingface dataset:
Saliency Prediction in the Deep Learning Era: Successes and Limitations

Visual saliency models have enjoyed a big leap in performance in recent years, thanks to advances in deep learning and large scale annotated data. Despite enormous effort and huge breakthroughs, however, models still fall short in reaching human-level accuracy. In this work, I explore the landscape of the field emphasizing on new deep saliency models, benchmarks, and datasets. A large number of image and video saliency models are reviewed and compared over two image benchmarks and two large scale video datasets. Further, I identify factors that contribute to the gap between models and humans and discuss the remaining issues that need to be addressed to build the next generation of more powerful saliency models. Some specific questions that are addressed include: in what ways current models fail, how to remedy them, what can be learned from cognitive studies of attention, how explicit saliency judgments relate to fixations, how to conduct fair model comparison, and what are the emerging applications of saliency models.
Adversarial learning for mono- or multi-modal registration

This paper introduces an unsupervised adversarial similarity network for image registration. Unlike existing deep learning registration methods, our approach can train a deformable registration network without the need of ground-truth deformations and specific similarity metrics. We connect a registration network and a discrimination network with a deformable transformation layer. The registration network is trained with the feedback from the discrimination network, which is designed to judge whether a pair of registered images are sufficiently similar. Using adversarial training, the registration network is trained to predict deformations that are accurate enough to fool the discrimination network. The proposed method is thus a general registration framework, which can be applied for both mono-modal and multi-modal image registration. Experiments on four brain MRI datasets and a multi-modal pelvic image dataset indicate that our method yields promising registration performance in accuracy, efficiency and generalizability compared with state-of-the-art registration methods, including those based on deep learning.
XRAI: Better Attributions Through Regions

Saliency methods can aid understanding of deep neural networks. Recent years have witnessed many improvements to saliency methods, as well as new ways for evaluating them. In this paper, we 1) present a novel region-based attribution method, XRAI, that builds upon integrated gradients (Sundararajan et al. 2017), 2) introduce evaluation methods for empirically assessing the quality of image-based saliency maps (Performance Information Curves (PICs)), and 3) contribute an axiom-based sanity check for attribution methods. Through empirical experiments and example results, we show that XRAI produces better results than other saliency methods for common models and the ImageNet dataset.
Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading

Abstract We present a unified framework tackling two problems: class-specific 3D reconstruction from a single image, and generation of new 3D shape samples. These tasks have received considerable attention recently; however, most existing approaches rely on 3D supervision, annotation of 2D images with keypoints or poses, and/or training with multiple views of each object instance. Our framework is very general: it can be trained in similar settings to existing approaches, while also supporting weaker supervision. Importantly, it can be trained purely from 2D images, without pose annotations, and with only a single view per instance. We employ meshes as an output representation, instead of voxels used in most prior work. This allows us to reason over lighting parameters and exploit shading information during training, which previous 2D-supervised methods cannot. Thus, our method can learn to generate and reconstruct concave object classes. We evaluate our approach in various settings, showing that: (i) it learns to disentangle shape from pose and lighting; (ii) using shading in the loss improves performance compared to just silhouettes; (iii) when using a standard single white light, our model outperforms state-of-the-art 2D-supervised methods, both with and without pose supervision, thanks to exploiting shading cues; (iv) performance improves further when using multiple coloured lights, even approaching that of state-of-the-art 3D-supervised methods; (v) shapes produced by our model capture smooth surfaces and fine details better than voxel-based approaches; and (vi) our approach supports concave classes such as bathtubs and sofas, which methods based on silhouettes cannot learn.
An analytical study of information extraction from unstructured and multidimensional big data

Abstract Process of information extraction (IE) is used to extract useful information from unstructured or semi-structured data. Big data arise new challenges for IE techniques with the rapid growth of multifaceted also called as multidimensional unstructured data. Traditional IE systems are inefficient to deal with this huge deluge of unstructured big data. The volume and variety of big data demand to improve the computational capabilities of these IE systems. It is necessary to understand the competency and limitations of the existing IE techniques related to data pre-processing, data extraction and transformation, and representations for huge volumes of multidimensional unstructured data. Numerous studies have been conducted on IE, addressing the challenges and issues for different data types such as text, image, audio and video. Very limited consolidated research work have been conducted to investigate the task-dependent and task-independent limitations of IE covering all data types in a single study. This research work address this limitation and present a systematic literature review of state-of-the-art techniques for a variety of big data, consolidating all data types. Recent challenges of IE are also identified and summarized. Potential solutions are proposed giving future research directions in big data IE. The research is significant in terms of recent trends and challenges related to big data analytics. The outcome of the research and recommendations will help to improve the big data analytics by making it more productive.
TexturePose: Supervising Human Mesh Estimation With Texture Consistency

This work addresses the problem of model-based human pose estimation. Recent approaches have made significant progress towards regressing the parameters of parametric human body models directly from images. Because of the absence of images with 3D shape ground truth, relevant approaches rely on 2D annotations or sophisticated architecture designs. In this work, we advocate that there are more cues we can leverage, which are available for free in natural images, i.e., without getting more annotations, or modifying the network architecture. We propose a natural form of supervision, that capitalizes on the appearance constancy of a person among different frames (or viewpoints). This seemingly insignificant and often overlooked cue goes a long way for model-based pose estimation. The parametric model we employ allows us to compute a texture map for each frame. Assuming that the texture of the person does not change dramatically between frames, we can apply a novel texture consistency loss, which enforces that each point in the texture map has the same texture value across all frames. Since the texture is transferred in this common texture map space, no camera motion computation is necessary, or even an assumption of smoothness among frames. This makes our proposed supervision applicable in a variety of settings, ranging from monocular video, to multi-view images. We benchmark our approach against strong baselines that require the same or even more annotations that we do and we consistently outperform them. Simultaneously, we achieve state-of-the-art results among model-based pose estimation approaches in different benchmarks. The project website with videos, results, and code can be found at https://seas.upenn.edu/~pavlakos/projects/texturepose.
Learning imbalanced datasets based on SMOTE and Gaussian distribution

The learning of imbalanced datasets is a ubiquitous challenge for researchers in the fields of data mining and machine learning. Conventional classifiers are often biased towards the majority class, and loss functions attempt to optimize the quantities. In this paper, we present two effective sampling methods that improve the data distributions. One rebalanced method, the Adaptive-SMOTE, improves the SMOTE method by adaptively selecting groups of Inner and Danger data from the minority class such that a new minority class is compiled based on the selected data, thus preventing an expansion of the category boundary and strengthening the distributional characteristics of the original data. The other method, Gaussian Oversampling, combines dimension reduction with the Gaussian distribution, which makes the tail of the Gaussian distribution thinner. Cross-validation experiments on 15 datasets show that the two sampling methods achieve significant improvements compared with other typical methods. The Adaptive-SMOTE has higher F-measure and Acc values than other existing sampling methods and higher robustness to classifiers and datasets with different values of Imb. Gaussian Oversampling is more efficient when dealing with extremely imbalanced classifications.
Teechain

Blockchains such as Bitcoin and Ethereum execute payment transactions securely, but their performance is limited by the need for global consensus. Payment networks overcome this limitation through off-chain transactions. Instead of writing to the blockchain for each transaction, they only settle the final payment balances with the underlying blockchain. When executing off-chain transactions in current payment networks, parties must access the blockchain within bounded time to detect misbehaving parties that deviate from the protocol. This opens a window for attacks in which a malicious party can steal funds by deliberately delaying other parties' blockchain access and prevents parties from using payment networks when disconnected from the blockchain.
Construction of the Virtual Embodiment Questionnaire (VEQ)

User embodiment is important for many virtual reality (VR) applications, for example, in the context of social interaction, therapy, training, or entertainment. However, there is no validated instrument to empirically measure the perception of embodiment, necessary to reliably evaluate this important quality of user experience (UX). To assess components of virtual embodiment in a valid, reliable, and consistent fashion, we develped a Virtual Embodiment Questionnaire (VEQ). We reviewed previous literature to identify applicable constructs and items, and performed a confirmatory factor analysis (CFA) on the data from three experiments (N = 196). Each experiment modified a distinct simulation property, namely, the level of immersion, the level of personalization, and the level of behavioral realism. The analysis confirmed three factors: (1) ownership of a virtual body, (2) agency over a virtual body, and (3) change in the perceived body schema. A fourth study (N = 22) further confirmed the reliability and validity of the scale and investigated the impacts of latency jitter of avatar movements presented in the simulation compared to linear latencies and a baseline. We present the final scale and further insights from the studies regarding related constructs.
VTNFP: An Image-Based Virtual Try-On Network With Body and Clothing Feature Preservation

Image-based virtual try-on systems with the goal of transferring a desired clothing item onto the corresponding region of a person have made great strides recently, but challenges remain in generating realistic looking images that preserve both body and clothing details. Here we present a new virtual try-on network, called VTNFP, to synthesize photo-realistic images given the images of a clothed person and a target clothing item. In order to better preserve clothing and body features, VTNFP follows a three-stage design strategy. First, it transforms the target clothing into a warped form compatible with the pose of the given person. Next, it predicts a body segmentation map of the person wearing the target clothing, delineating body parts as well as clothing regions. Finally, the warped clothing, body segmentation map and given person image are fused together for fine-scale image synthesis. A key innovation of VTNFP is the body segmentation map prediction module, which provides critical information to guide image synthesis in regions where body parts and clothing intersects, and is very beneficial for preventing blurry pictures and preserving clothing and body part details. Experiments on a fashion dataset demonstrate that VTNFP generates substantially better results than state-of-the-art methods.
Language-Conditioned Graph Networks for Relational Reasoning

Solving grounded language tasks often requires reasoning about relationships between objects in the context of a given task. For example, to answer the question "What color is the mug on the plate?" we must check the color of the specific mug that satisfies the "on" relationship with respect to the plate. Recent work has proposed various methods capable of complex relational reasoning. However, most of their power is in the inference structure, while the scene is represented with simple local appearance features. In this paper, we take an alternate approach and build contextualized representations for objects in a visual scene to support relational reasoning. We propose a general framework of Language-Conditioned Graph Networks (LCGN), where each node represents an object, and is described by a context-aware representation from related objects through iterative message passing conditioned on the textual input. E.g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction. We experimentally show that our LCGN approach effectively supports relational reasoning and improves performance across several tasks and datasets. Our code is available at http://ronghanghu.com/lcgn.
Analysis of Data Management in Blockchain-Based Systems: From Architecture to Governance

In a blockchain-based system, data and the consensus-based process of recording and updating them over distributed nodes are central to enabling the trustless multi-party transactions.Thus, properly understanding what and how the data are stored and manipulated ultimately determines the degree of utility, performance, and cost of a blockchain-based application.While blockchains enhance the quality of the data by providing a transparent, immutable, and consistent data store, the technology also brings new challenges from a data management perspective.In this paper, we analyse blockchains from the viewpoint of a developer to highlight important concepts and considerations when incorporating a blockchain into a larger software system as a data store.The work aims to increase the level of understanding of blockchain technology as a data store and to promote a methodical approach in applying it to large software systems.First, we identify the common architectural layers of a typical software system with data stores and conceptualise each layer in blockchain terms.Second, we examine the placement and flow of data in blockchain-based applications.Third, we explore data administration aspects for blockchains, especially as a distributed data store.Fourth, we discuss the analytics of blockchain data and trustable data analytics enabled by blockchain.Lastly, we examine the data governance issues in blockchains in terms of privacy and quality assurance.
An Automatic Nucleus Segmentation and CNN Model based Classification Method of White Blood Cell

Abstract White blood cells (WBCs) play a remarkable role in the human immune system. To diagnose blood-related diseases, pathologists need to consider the characteristics of WBC. The characteristics of WBC can be defined based on the morphological properties of WBC nucleus. Therefore, nucleus segmentation plays a vital role to classify the WBC image and it is an important part of the medical diagnosis system. In this study, color space conversion and k-means algorithm based new WBC nucleus segmentation method is proposed. Then we localize the WBC based on the location of segmented nucleus to separate them from the entire blood smear image. To classify the localized WBC image, we propose a new convolutional neural network (CNN) model by combining the concept of fusing the features of first and last convolutional layers, and propagating the input image to the convolutional layer. We also use a dropout layer for preventing the model from overfitting problem. We show the effectiveness of our proposed nucleus segmentation method by evaluating with seven quality metrics and comparing with other methods on four public databases. We achieve average accuracy of 98.61% and more than 97% on each public database. We also evaluate our proposed CNN model by using nine classification metrics and achieve an overall accuracy of 96% on BCCD test database. To validate the generalization capability of our proposed CNN model, we show the training and testing accuracy and loss curves for random test set of BCCD database. Further, we compare the performance of our proposed CNN model with four state-of-the-art CNN models (biomedical image classifier) by measuring the value of evaluation metrics.
TableNet: Deep Learning Model for End-to-end Table Detection and Tabular Data Extraction from Scanned Document Images

With the widespread use of mobile phones and scanners to photograph and upload documents, the need for extracting the information trapped in unstructured document images such as retail receipts, insurance claim forms and financial invoices is becoming more acute. A major hurdle to this objective is that these images often contain information in the form of tables and extracting data from tabular sub-images presents a unique set of challenges. This includes accurate detection of the tabular region within an image, and subsequently detecting and extracting information from the rows and columns of the detected table. While some progress has been made in table detection, extracting the table contents is still a challenge since this involves more fine grained table structure (rows & columns) recognition. Prior approaches have attempted to solve the table detection and structure recognition problems independently using two separate models. In this paper, we propose TableNet: a novel end-to-end deep learning model for both table detection and structure recognition. The model exploits the interdependence between the twin tasks of table detection and table structure recognition to segment out the table and column regions. This is followed by semantic rule-based row extraction from the identified tabular sub-regions. The proposed model and extraction approach was evaluated on the publicly available ICDAR 2013 and Marmot Table datasets obtaining state of the art results. Additionally, we demonstrate that feeding additional semantic features further improves model performance and that the model exhibits transfer learning across datasets. Another contribution of this paper is to provide additional table structure annotations for the Marmot data, which currently only has annotations for table detection.
Understanding Distributed Poisoning Attack in Federated Learning

Federated learning is inherently vulnerable to poisoning attacks, since no training samples will be released to and checked by trustworthy authority. Poisoning attacks are widely investigated in centralized learning paradigm, however distributed poisoning attacks, in which more than one attacker colludes with each other, and injects malicious training samples into local models of their own, may result in a greater catastrophe in federated learning intuitively. In this paper, through real implementation of a federated learning system and distributed poisoning attacks, we obtain several observations about the relations between the number of poisoned training samples, attackers, and attack success rate. Moreover, we propose a scheme, Sniper, to eliminate poisoned local models from malicious participants during training. Sniper identifies benign local models by solving a maximum clique problem, and suspected (poisoned) local models will be ignored during global model updating. Experimental results demonstrate the efficacy of Sniper. The attack success rates are reduced to around 2% even a third of participants are attackers.
A perspective on multiparameter quantum metrology: From theoretical tools to applications in quantum imaging

The interest in a system often resides in the interplay among different parameters governing its evolution. It is thus often required to access many of them at once for a complete description. Assessing how quantum enhancement in such multiparameter estimation can be achieved depends on understanding the many subtleties that come into play: establishing solid foundations is key to delivering future technology for this task. In this article we discuss the state of the art of quantum multiparameter estimation, with a particular emphasis on its theoretical tools, on application to imaging problems, and on the possible avenues towards the next developments.
Balancing composite motion optimization

Meta-heuristic algorithms play an important role in the optimization field thanks to their robustness and programming simplicity. Many meta-heuristic methods have been devised in recent years. Inspired by nature, they usually simulate natural or human-specific phenomena in a better way. A large amount of them are based on complicated behaviors requiring several implementation steps and algorithm-specific control parameters, which impedes users and limits solutions to different types of optimization problems. Hence, designing effective simple and parameter-free optimization methods attracts much attention. In this paper, we propose a novel population-based optimization algorithm based on balancing composite motions (BCMO). The core idea is balancing composite motion properties of individuals in solution space. Equalizing global and local searches via a probabilistic selection model creates a movement mechanism of each individual. Four test suites selected in the literature, which vary from numerical benchmarks to practical problems, to demonstrate the performance of BCMO include: (1) 23 classical benchmark functions, (2) CEC 2005 benchmark functions, (3) CEC 2014 benchmark functions, and (4) 3 real engineering design problems. The statistical results reveal the promising performance and application of BCMO in a variety of optimization and practical problems with constrained and unknown search spaces.