Methods for surface understanding from multi-modal imagery
Creators: name: Purri, Matthew
Publisher: name: No Publisher Supplied
Year: 2022

description: A fundamental goal within the field of computer vision is to develop a system capable of understanding all aspects of a given visual input. Since the (re)emergence of convolutional neural networks, performance on object-centric vision tasks such as image classification and object detection has surpassed many expectations. However, less progress and effort has been dedicated to understanding material and texture components of visual media. Material and texture understanding can enhance applications such as autonomous vehicles avoiding dangerous driving surfaces and high-resolution object reconstruction. This thesis is dedicated to both creating pragmatic and informative surface-centric datasets and developing robust surface representations for machine learning models.We begin by exploring how to estimate the quantitative properties of a surface from images. Specifically, we focus on deriving tactile properties from natural images. The human sensory system learns from a young age to map visual stimuli to perceived tactile responses. We aim to develop a similar system, mapping images of surfaces to numerical tactile estimates. Since machine learning models require many examples, we collect the largest visual-tactile dataset comprised of multi-view surface images and corresponding tactile property measurements. We conduct an empirical study to answer the question: what tactile properties can be reliably estimated from an image? Our results show that 6% of the tactile properties we measured can be estimated from a single view of a surface and with further model development we raise it to 93%. Next, we consider if there is a benefit to using more than one image of a surface as input and if so, how to utilize this information within a machine learning system. Compared to various subsampling techniques and utilizing all available images, our proposed learned select-view network improves the coefficient of determination (R²) and %ₑᵣᵣ by 10% and 8% compared to the best single-view model.Next we investigate several situations for estimating the geometry of objects based on surface understanding. High quality outdoor mono-view image-based surface reconstruction methods require image captures over the coarse of many days. We are the first to take advantage of the naturally occurring structured illumination polarimetric pattern of the sky, to develop a method that requires much less time (instantly to a couple of hours) and creates more precise reconstructions. Our method compares the observed polarimetric reflection of the surface against an initial model, then iteratively solves for global and surface normal parameters until the observed and derived models converge. Additionally, we explore a more constrained scenario where we train a model to estimate the geometry of a rotating object from a time-series monostatic radar signal. Experiments are conducted to examine the robustness of the model and we develop a learned synthetic generation method to increase the realism of synthetic input.Finally, we study how to represent surface information within a machine learning system. Given an irregular number of observations of a surface at random viewing and illumination directions, how can we represent this information in a concise manner for a convolutional neural network? We develop a material reflectance-based encoding that compactly represents observations as the residual between the observations and a set of pre-measured dense material reflectances. Our representation results improve material semantic segmentation performance by 5% absolute mIoU over baseline models. Next, we use the temporal consistency of aligned satellite imagery to learn a robust surface representation without labels. Using a discretization bottleneck and a reconstruction objective, we learn surface classes for each pixel. With this robust characterization, we train a model to learn the difference between natural and artificial material state transitions over time to predict change within a scene. Additionally, we investigate generating synthetic change video sequences based using the learned surface classes. Our proposed network results in state-of-the-art performance on several multi-temporal change detection datasets.