Reviewerly/exHarmony huggingface dataset:
Another step toward demystifying deep neural networks

The field of deep learning has positioned itself in the past decade as a prominent and extremely fruitful engineering discipline. This comeback of neural networks in the early 2000s swept the machine learning community, and soon after found itself immersed in practically every scientific, social, and technological front. A growing series of contributions established this field as leading to state-of-the-art results in nearly every task, recognizing image content, understanding written documents, exposing obscure connections in massive datasets, facilitating efficient search in large repositories, translating languages, enabling a revolution in transportation, revealing new scientific laws in physics and chemistry, and so much more. Deep neural networks not only solve known problems but offer, in addition, unprecedented results in deploying learning to problems that until recently were considered as hopeless or only weakly successful. These include automatically synthesizing text–media, creating musical art pieces, synthesizing realistic images and video, enabling competitive game-playing, and this list goes on and on. Amazingly, all these great empirical achievements are obtained with hardly any theoretical foundations that could provide a clear justification for the architectures used, an understanding of the algorithms that accompany them, a clear mathematical reasoning behind the various tricks employed, and above all, the impressive results obtained. The quest for a theory that could explain these ingredients has become the Holy Grail of data sciences. Various impressive attempts to provide such a theory have started to appear, relying on ideas from various disciplines (see, e.g., refs. 1⇓⇓⇓⇓⇓⇓⇓–9). The paper by Papyan et al. (10) in PNAS adds an important layer to this vast attempt of developing a comprehensive theory that explains the behavior of deep learning solutions. In this Commentary, we provide a wide context to their results, highlight and clarify their contribution, and raise … [↵][1]1To whom correspondence may be addressed. Email: elad{at}cs.technion.ac.il. [1]: #xref-corresp-1-1
An Automated Pipeline for a Browser-based, City-scale Mobile 4D VR Application based on Historical Images

The process for automatically creating 3D city models from contemporary photographs and visualizing them on mobile devices is now well established, but historical 4D city models are more challenging. The fourth dimension here is time. This article describes an automated VR pipeline based on historical photographs and resulting in an interactive browser-based device-rendered 4D visualization and information system for mobile devices. Since the pipeline shown is currently still under development, initial results for stages of the process will be shown and assessed for accuracy and usability.
Determination of fracture toughness of polymer coating using micro-scale digital image correlation technique

This paper presents a comprehensive study on determining the interface adhesive strength of the polymer coating/substrate system to improve its durability. Interfacial stress intensity factors based on the crack tip opening displacement (CTOD) method were theoretically analysed and verified by the finite element simulation. Thereafter, an experimental setup was established with the micro-scale digital image correlation (DIC) method to measure the CTODs and the designed loading fixture for multi-angle loading. The apparent specimen rotation and out-of-plane displacement were significantly restrained in the test, and the measured displacement accuracy of the micro-scale DIC method was approximately 1/10 of the minimum CTOD value. Mixed-mode stress intensity factors and an interfacial failure criterion curve in terms of stress intensity factors were obtained. Furthermore, the test method was applied to evaluate the interfacial fracture toughness of nanoparticle-reinforced epoxy coatings. The experimental results demonstrate that determining the stress intensity factors on the interface of the polymer coating/substrate system based on the CTODs is an effective and reliable approach.
Feature selection and embedding based cross project framework for identifying crashing fault residence

Context: The automatically produced crash reports are able to analyze the root of fault causing the crash (crashing fault for short) which is a critical activity for software quality assurance. Objective: Correctly predicting the existence of crashing fault residence in stack traces of crash report can speed up program debugging process and optimize debugging efforts. Existing work focused on the collected label information from bug-fixing logs, and the extracted features of crash instances from stack traces and source code for Identification of Crashing Fault Residence (ICFR) of newly-submitted crashes. This work develops a novel cross project ICFR framework to address the data scarcity problem by using labeled crash data of other project for the ICFR task of the project at hand. This framework removes irrelevant features, reduces distribution differences, and eases the class imbalance issue of cross project data since these factors may negatively impact the ICFR performance. Method: The proposed framework, called FSE, combines Feature Selection and feature Embedding techniques. The FSE framework first uses an information gain ratio based feature ranking method to select a relevant feature subset for cross project data, and then employs a state-of-the-art Weighted Balanced Distribution Adaptation (WBDA) method to map features of cross project data into a common space. WBDA considers both marginal and conditional distributions as well as their weights to reduce data distribution discrepancies. Besides, WBDA balances the class proportion of each project data to alleviate the class imbalance issue. Results: We conduct experiments on 7 projects to evaluate the performance of our FSE framework. The results show that FSE outperforms 25 methods under comparison. Conclusion: This work proposes a cross project learning framework for ICFR, which uses feature selection and embedding to remove irrelevant features and reduce distribution differences, respectively. The results illustrate the performance superiority of our FSE framework.
Application of multi-component gas geochemical survey for deep mineral exploration in covered areas

There are many types of coverage areas in China, which are widely distributed and have huge resource potential. The overburden materials (vegetation, slope deposit, sedimentary rock, volcanic rocks, etc.) in coverage areas have introduced great difficulties and challenges to mineral exploration. In this contribution, a Portable Multi-component Gas Rapid Analyzer (PMGRA) was developed, a new instrument characterized by fast analysis, high precision, low cost, and easy portability. It can simultaneously detect a variety of mineralization-related gases (H2S, SO2, CH4, and CO2). Using PMGRA, accurate soil-gas geochemical survey was conducted in the Weilasituo Sn polymetallic deposit and Zhugongtang PbZn deposit. These hydrothermal deposits are dominated by faults and ores abundant in sulfide. The experimental results showed that PMGRA can detect accurate gas concentration data in the sediments of two covered areas. Above the deep-buried ore bodies, H2S, SO2, CH4, and CO2 gases had strong anomalies, and there was obviously strong gas anomaly in the upper part of the fault. It indicated that the gas geochemical measurement method using the gas rapid analyzer can reveal effective information about deep-buried ore bodies and faults. In the upper plate area of the fault, the peak of CO2 usually appeared before the peaks the H2S, SO2, and CH4, which could be used to predict the trend of the fault. The experiment also showed that the method can detect the mineralization information at 500 m and 800 m in the Weilasituo tin polymetallic deposit and the Zhugongtang PbZn deposit, indicating that the method has large detection depth. Therefore, the gas geochemical survey based on PMGRA is an effective working method suitable for mineral exploration in covered areas, which can provide a new idea for prospecting concealed ores in covered areas.
Revealing True Identity: Detecting Makeup Attacks in Face-based Biometric Systems

Face-based authentication systems are among the most commonly used biometric systems, because of the ease of capturing face images at a distance and in non-intrusive way. These systems are, however, susceptible to various presentation attacks, including printed faces, artificial masks, and makeup attacks. In this paper, we propose a novel solution to address makeup attacks, which are the hardest to detect in such systems because makeup can substantially alter the facial features of a person, including making them appear older/younger by adding/hiding wrinkles, modifying the shape of eyebrows, beard, and moustache, and changing the color of lips and cheeks. In our solution, we design a generative adversarial network for removing the makeup from face images while retaining their essential facial features and then compare the face images before and after removing makeup. We collect a large dataset of various types of makeup, especially malicious makeup that can be used to break into remote unattended security systems. This dataset is quite different from existing makeup datasets that mostly focus on cosmetic aspects. We conduct an extensive experimental study to evaluate our method and compare it against the state-of-the art using standard objective metrics commonly used in biometric systems as well as subjective metrics collected through a user study. Our results show that the proposed solution produces high accuracy and substantially outperforms the closest works in the literature.
Quasi-Probability Husimi-Distribution Information and Squeezing in a Qubit System Interacting with a Two-Mode Parametric Amplifier Cavity

Squeezing and phase space coherence are investigated for a bimodal cavity accommodating a two-level atom. The two modes of the cavity are initially in the Barut–Girardello coherent states. This system is studied with the SU(1,1)-algebraic model. Quantum effects are analyzed with the Husimi function under the effect of the intrinsic decoherence. Squeezing, quantum mixedness, and the phase information, which are affected by the system parameters, exalt a richer structure dynamic in the presence of the intrinsic decoherence.
Investigating the Impact of Energy Source Level on the Self-Guided Vehicle System Performances, in the Industry 4.0 Context

Automated industrial vehicles are taking an imposing place by transforming the industrial operations, and contributing to an efficient in-house transportation of goods. They are expected to bring a variety of benefits towards the Industry 4.0 transition. However, Self-Guided Vehicles (SGVs) are battery-powered, unmanned autonomous vehicles. While the operating durability depends on self-path design, planning energy-efficient paths become crucial. Thus, this paper has no concrete contribution but highlights the lack of energy consideration of SGV-system design in literature by presenting a review of energy-constrained global path planning. Then, an experimental investigation explores the long-term effect of battery level on navigation performance of a single vehicle. This experiment was conducted for several hours, a deviation between the global trajectory and the ground-true path executed by the SGV was observed as the battery depleted. The results show that the mean square error (MSE) increases significantly as the battery’s state-of-charge decreases below a certain value.
II-20: Intelligent and pragmatic analytic categorization of image collections

In this paper, we introduce 11-20 (Image Insight 2020), a multimedia analytics approach for analytic categorization of image collections. Advanced visualizations for image collections exist, but they need tight integration with a machine model to support the task of analytic categorization. Directly employing computer vision and interactive learning techniques gravitates towards search. Analytic categorization, however, is not machine classification (the difference between the two is called the pragmatic gap): a human adds/redefines/deletes categories of relevance on the fly to build insight, whereas the machine classifier is rigid and non-adaptive. Analytic categorization that truly brings the user to insight requires a flexible machine model that allows dynamic sliding on the exploration-search axis, as well as semantic interactions: a human thinks about image data mostly in semantic terms. 11-20 brings three major contributions to multimedia analytics on image collections and towards closing the pragmatic gap. Firstly, a new machine model that closely follows the user's interactions and dynamically models her categories of relevance. II-20's machine model, in addition to matching and exceeding the state of the art's ability to produce relevant suggestions, allows the user to dynamically slide on the exploration-search axis without any additional input from her side. Secondly, the dynamic, 1-image-at-a-time Tetris metaphor that synergizes with the model. It allows a well-trained model to analyze the collection by itself with minimal interaction from the user and complements the classic grid metaphor. Thirdly, the fast-forward interaction, allowing the user to harness the model to quickly expand (“fast-forward”) the categories of relevance, expands the multimedia analytics semantic interaction dictionary. Automated experiments show that II-20's machine model outperforms the existing state of the art and also demonstrate the Tetris metaphor's analytic quality. User studies further confirm that II-20 is an intuitive, efficient, and effective multimedia analytics tool.
A Curvature-Enhanced Random Walker Segmentation Method for Detailed Capture of 3D Cell Surface Membranes

High-resolution 3D microscopy is a fast advancing field and requires new techniques in image analysis to handle these new datasets. In this work, we focus on detailed 3D segmentation of Dictyostelium cells undergoing macropinocytosis captured on an iSPIM microscope. We propose a novel random walker-based method with a curvature-based enhancement term, with the aim of capturing fine protrusions, such as filopodia and deep invaginations, such as macropinocytotic cups, on the cell surface. We tested our method on both real and synthetic 3D image volumes, demonstrating that the inclusion of the curvature enhancement term can improve the segmentation of the aforementioned features. We show that our method performs better than other state of the art segmentation methods in 3D images of Dictyostelium cells, and performs competitively against CNN-based methods in two Cell Tracking Challenge datasets, demonstrating the ability to obtain accurate segmentations without the requirement of large training datasets. We also present an automated seeding method for microscopy data, which, combined with the curvature-enhanced random walker method, enables the segmentation of large time series with minimal input from the experimenter.
Registration of multimodal images with edge features and scale invariant PIIFD

For the past few decades, due to the extensive application of multi-sensor vision systems, the technology of multimodal image registration has been continuously developed, especially the registration of infrared and visible images. However, due to the difference of grayscale distribution, the results of most existing methods for infrared and visible image registration are commonly not satisfactory. Therefore, this paper proposes a method for visible and infrared image registration using edge features and partial intensity invariant feature descriptors (PIIFD) with scale invariance. This method extracts conspicuous edge features of the source image pair through a window grayscale weight algorithm, and calculates the feature descriptions of the edge images with scale invariant PIIFD. And then Gaussian field estimator and affine transform are used to realize image feature matching and alignment transformation. The quality and quantity comparisons with 5 other most advanced methods reveal that our method can attain an ascendant performance, can accurately and efficiently realize the registration of infrared and visible images.
Bridging the Gap between Vision and Language Domains for Improved Image Captioning

Image captioning has attracted extensive research interests in recent years. Due to the great disparities between vision and language, an important goal of image captioning is to link the information in visual domain to textual domain. However, many approaches conduct this process only in the decoder, making it hard to understand the images and generate captions effectively. In this paper, we propose to bridge the gap between the vision and language domains in the encoder, by enriching visual information with textual concepts, to achieve deep image understandings. To this end, we propose to explore the textual-enriched image features. Specifically, we introduce two modules, namely Textual Distilling Module and Textual Association Module. The former distills relevant textual concepts from image features, while the latter further associates extracted concepts according to their semantics. In this manner, we acquire textual-enriched image features, which provide clear textual representations of image under no explicit supervision. The proposed approach can be used as a plugin and easily embedded into a wide range of existing image captioning systems. We conduct the extensive experiments on two benchmark image captioning datasets, i.e., MSCOCO and Flickr30k. The experimental results and analysis show that, by incorporating the proposed approach, all baseline models receive consistent improvements over all metrics, with the most significant improvement up to 10% and 9%, in terms of the task-specific metrics CIDEr and SPICE, respectively. The results demonstrate that our approach is effective and generalizes well to a wide range of models for image captioning.
Objectness Consistent Representation for Weakly Supervised Object Detection

Weakly supervised object detection aims at learning object detectors with only image-level category labels. Most existing methods tend to solve this problem by using a multiple instance learning detector which is usually trapped to discriminate object parts. In order to select high-quality proposals, recent works leverage objectness scores derived from weakly-supervised segmentation maps to rank the object proposals. Base on our observation, this kind of segmentation guided method always fails due to neglect of the fact that the objectness of all proposals inside the ground-truth box should be consistent. In this paper, we propose a novel object representation named Objectness Consistent Representation (OCRepr) to meet the consistency criterion of objectness. Specifically, we project the segmentation confidence scores into two orthogonal directions, namely vertical and horizontal, to get the OCRepr. With the novel object representation, more high-quality proposals can be mined for learning a much stronger object detector. We obtain 54.6% and 51.1% mAP scores on VOC 2007 and 2012 datasets, significantly outperforming the state-of-the-art and demonstrating the superiority of OCRepr for weakly supervised object detection.
Information-Theoretic Bounds on Transfer Generalization Gap Based on Jensen-Shannon Divergence

In transfer learning, training and testing data sets are drawn from different data distributions. The transfer generalization gap is the difference between the population loss on the target data distribution and the training loss. The training data set generally includes data drawn from both source and target distributions. This work presents novel information-theoretic upper bounds on the average transfer generalization gap that capture ( <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$i$</tex> ) the domain shift between the target data distribution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P_{Z}^{\prime}$</tex> and the source distribution <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P_{Z}$</tex> through a two-parameter family of generalized <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\alpha_{1},\ \alpha_{2})$</tex> -Jensen-Shannon (JS) divergences; and (ii) the sensitivity of the transfer learner output <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$W$</tex> to each individual sample of the data set <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Z_{i}$</tex> via the mutual information <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$I(W;Z_{i})$</tex> . For <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\alpha_{1}\in(0,1)$</tex> , the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\alpha_{1},\ \alpha_{2})$</tex> - JS divergence can be bounded even when the support of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P_{Z}$</tex> is not included in that of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$P_{Z}^{\prime}$</tex> . This contrasts the Kullback-Leibler (KL) divergence <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$D_{\text{KL}}(P_{Z}\Vert P_{Z}^{\prime})$</tex> -based bounds of Wu et al. [1], which are vacuous under this assumption. Moreover, the obtained bounds hold for unbounded loss functions with bounded cumulant generating functions, unlike the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\phi$</tex> -divergence based bound of Wu et al. [1]. We also obtain new upper bounds on the average transfer excess risk in terms of the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\alpha_{1},\ \alpha_{2})$</tex> -JS divergence for empirical weighted risk minimization (EWRM), which minimizes the weighted average training losses over source and target data sets. Finally, we provide a numerical example to illustrate the merits of the introduced bounds.
Which Model to Transfer? Finding the Needle in the Growing Haystack

Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular for computer vision tasks where it provides a remarkably solid baseline. The emergence of rich model repositories, such as TensorFlow Hub, enables the practitioners and researchers to unleash the potential of these models across a wide range of downstream tasks. As these repositories keep growing exponentially, efficiently selecting a good model for the task at hand becomes paramount. We provide a formalization of this problem through afamiliar notion of regret and introduce the predominant strategies, namely task-agnostic (e.g. ranking models by their ImageNet performance) and task-aware search strategies (such as linear or kNN evaluation). We conduct a large-scale empirical study and show that both task-agnostic and task-aware methods can yield high regret. We then propose a simple and computationally efficient hybrid search strategy which outperforms the existing approaches. We highlight the practical benefits of the proposed solution on a set of 19 diverse vision tasks.