Towards Generic Interface for Human-Neural Network Knowledge Exchange

1 INTRODUCTION . From medicine ( Shen et al. , 2017 ) to self-driving cars ( Buczak & Guven , 2015 ) , machine learning ( ML ) systems are increasingly ubiquitous and outperform humans in multiple tasks . Given the breadth and importance of ML applications , it is critical for humans to use modern machine learning ( ML ) models , such as neural networks ( NN ) , safely and trustfully , and to avoid unpredictable mistakes ( Zerilli et al. , 2019 ) . This requires that humans should be involved in the ML loop . Thus , an interface between human and NN is needed , where NN and human can easily understand , interact , and influence each other . Especially when NN makes mistakes , human prior and domain expertise knowledge with causal inference ability and common sense may help NN achieve better performance . However , the lack of an effective interface makes it hard for a human to locate the reason for NN ’ s error , not to mention correct the error . There are two main challenges for the interaction between humans and NN : ( 1 ) Interpretability , for humans , i.e. , how to understand the reasoning logic of NN and how to correctly locate the reason of errors . ( 2 ) Changing NN ’ s logic and decision , once humans locate the error of NN , i.e. , how to correct it and improve NN ’ s performance ( Fig . 1 ) . To interpret a NN , current pixel-level interpretation methods ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) are limited to low-level relationships . Even recent human-intuitive concept-level explanations ( Ghorbani et al. , 2019a ; Kim et al. , 2018 ) do not reveal the reasoning logic of NN . Visual reasoning explanation ( Ge et al. , 2021 ) mimics the reasoning logic of original NN and provide logical , easy-tounderstand explanations for final decisions , but can not directly influence NN ’ s performance to achieve closed-loop interaction . To interact with NN , current Human-In-the-Loop ML and Interacting ML are ML-centered methods that revolve around a pipeline of re-training a model using human-curated data instead of interacting with their reasoning logic ( Dellermann et al. , 2021 ) . To solve these problems , we propose Human-NN-Interface ( HNI ) for knowledge exchange , with the following key contributions : ( 1 ) HNI use high-level class-specific visual concepts and their relationships to build class-specific structural concepts graph ( c-SCG ) for each class of interest . The c-SCG is a description of the key parts ( concepts ; graph nodes ) of an object class and their spatial relationships ( graph edges ) . HNI allows human and NN to understand each other with c-SCG as a “ language ” to communicate , interact and exchange knowledge . ( 2 ) Through the NN-to-Human path , NN can use c-SCG to show their reasoning logic in a way that is intuitive to human understanding . ( 3 ) Through the Human-to-NN path , a human can analyze the reasoning logic ( c-SCG ) of NN and directly modify it with human prior knowledge in a very shot time ( around 13s to modify each class ) . Then , HNI can use Graph reasoning Network and partial knowledge distillation to transfer knowledge from human back to NN , such that the NN obtains new knowledge from humans . ( 4 ) By creating new c-SCGs or modifying existing c-SCGs , humans can “ teach ” NN about new objects they have never seen before , which provides a new pipeline to achieve zero-shot learning . 2 RELATED WORK . We review research areas that share similarities with our work , to position our contributions . Human-AI Interaction for Machine Learning ( ML ) applications aims to best combine human domain expertise and computational power of ML . To satisfy this need , Human-In-the-Loop ML ( HILML ) ( Dellermann et al. , 2021 ) and Interactive Machine Learning ( IML ) ( Ware et al. , 2002 ) have recently emerged . However , both are ML-centered methods which let humans play a “ server ” role around the ML process , from data production , ML modeling , to model evaluation and refinement ( Maadi et al. , 2021 ) . This limits human involvement and domain expert performance . “ Tell me where to look ” ( Li et al. , 2018 ) uses an explainable attention map to correct segmentation errors of NN , which is thus limited to low-level relationships . User interaction was also introduced in the image generation task , Interactive Image Generation ( Mittal et al. , 2019 ) can repeatedly modify images based on modifications to the scene graph while keeping the contents generated over previous steps . In our work , we take a step toward an interface with which human users and NN can more efficiently communicate , interact and exchange knowledge between each other , with no dependency on a given list of attributes . Interpreting neural networks . Research on interpretability methods for neural networks ( NNs ) has received more and more attention . Some try to explain NN by visualizing the correlation between each pixel of the input image and the final outputs . For instance , CAM ( Zhou et al. , 2016 ) and Grad-CAM ( Selvaraju et al. , 2017 ) can generate class-specific attention maps . Differently , several recent works focus on more human-intuitive concept-level explanations . ACE and TCAV ( Ghorbani et al. , 2019a ; Kim et al. , 2018 ) proposed algorithms to extract meaningful concepts from images and then produce an understandable explanation . VRX ( Ge et al. , 2021 ) uses visual concepts to explain an NN ’ s reasoning logic . Based on this work , our HNI goes beyond this and we treat logic explanation as a foundation , allowing humans to directly modify NN ’ s logic . Then HNI can transfer human ’ s knowledge back to the original NN to achieve a closed-loop interaction between humans and NN . Graph Neural Networks ( GNN ) are deep learning methods that operate on graph domains , which learn to represent graph nodes , edges , or subgraphs by low-dimensional vectors ( Zhou et al. , 2021 ) . Considering the trackable information-communication properties of GNNs , many reasoning tasks have adopted GNNs as a tool , such as VQA ( Teney et al. , 2017 ; Norcliffe-Brown et al. , 2018 ) , scene understanding ( Li et al. , 2017a ) , and semantic explanations ( Ge et al. , 2021 ) . In this work , we use a GNN-based Graph Reasoning Network ( GRN ) as the bridge between NNs and humans . It takes SCG as input and transfers knowledge with NN through knowledge distillation . Knowledge distillation can effectively learn a small student model from a large ensembled teacher model using soft targets ( Hinton et al. , 2015 ) . Knowledge distillation has been widely used in privileged learning ( Lopez-Paz et al. , 2015 ) , adversarial defense ( Papernot et al. , 2016 ) , learning with noisy data ( Li et al. , 2017b ) . In this work , we use knowledge distillation to transfer knowledge between a NN and a GNN-based graph reasoning network ( GRN ) which encodes human ’ s prior knowledge to achieve human-NN knowledge exchange . Zero-shot learning aims to train a classifier that can classify testing instances that belong to classes that are never seen before . Existing works include using the one-vs-rest solution ( Verma & Rai , 2017 ) , synthesizing pseudo instances of the unseen class ( Guo et al. , 2017 ) , projecting feature space instances and semantic space prototypes into a common space ( Palatucci et al. , 2009 ) , and using similar seen class as the positive instance of the unseen class ( Gan et al. , 2016 ) . Huynh & Elhamifar ( 2020 ) conduct compositional zero-shot learning by using a feature composition framework to extract and combine features of attributes to construct fine-grained attributes for unseen classes . Jia et al . ( 2021 ) use active zero-shot learning which promotes human-AI teaming by actively modifying the class-attribute matrix . However such attribute labels may not be always available in real-world scenarios . Here , we propose a new pipeline with HNI , which makes no assumption on the availability of attribute label , humans help build the understanding of the new class using learned visual concepts and structure . Then HNI transfers the knowledge of unseen class back to NN . 3 HUMAN-NN INTERFACE . Our proposed Human-NN Interface ( HNI ) to bridge the interaction between human and neural networks is visually summarized in Fig . 2 . There are two main path . 3.1 NN-TO-HUMAN . NN-to-human path explains the NN ’ s reasoning logic for each decision ( instance-level explanation , see Appendix Sec . A.1 ) and more importantly , the understanding of NN for each class , represented as class-specific Structural Concept Graph ( c-SCG ) . Each c-SCG is bound to one class ( Fig . 2 shows the c-SCG of school bus ) , where the nodes represent the important visual concepts that original NN considered most important in identifying the class of interest , and edges represent the pairwise structural relationships ( dependencies ) between concepts . As shown in Fig . 2 ( top ) , given a trained NN , there are two main steps in order to explain the reasoning logic of NN to human users : ( 1 ) Using Visual Concept Extractor ( VCE ) to discover representative visual concepts for each class of interest . The detailed procedure follows Appendix Fig . 8 ( a ) of ( Ge et al. , 2021 ) : To discover concepts for each class , we collect 50 to 100 images of the class . We first use top-down gradient attention ( Grad-Cam ( Selvaraju et al. , 2017 ) ) to constrain the relevant regions for concept proposals to the foreground segments , thereby ruling out irrelevant background patterns for this class . Then we follow the same workflow as the ACE paper ( Ghorbani et al. , 2019b ) : multi-resolution segmentation , feature extraction , clustering patches in latent space to obtain the concept candidates , and sorting these concept candidates based on importance score , similar to ( Kim et al. , 2018 ) . After that we obtain the concept pool ( each concept is represented by one mean feature vector , sorted by importance score ) for each class , which will serve as a source of concept candidates when human users modify concepts ( nodes ) for c-SCG . To build c-SCG that reveals the reasoning logic for each class of interest of the original NN , we select the top k ( k=4 in our experiments ) important concepts and their mean feature vectors as nodes . Each directed edge in an SCG edgeji = ( vj , vi ) has two attributes : 1 ) representation of the spatial structural relationship between nodes ; 2 ) dependency eji ( a trainable scalar ) between concepts vi , vj . The edge features can reveal the underlying causal logic of interactions between visual concepts which is crucial for the final decision . c-SCG extract such relationship between visual concepts during training and incorporate them as edge features in c-SCG . By showing c-SCG for each class of interest , NN-to-human path provides easy-to-understand insights for human users on the reasoning process of NN , which is also a foundation for the Human-to-NN path . The edges ( structural relationship and dependency ) are fully connected at the beginning and then , using the learning in the following step 2 , we select and only keep the important edges . ( 2 ) Using Graph Reasoning Network ( GRN ) to mimic the decision-making process of original NN with knowledge distillation . As a GNN-based network , GRN takes a graph as input . To train GRN based on the built c-SCGs ( one for each of n classes of interest ) , we need to establish connections between training images and the c-SCGs . To this end , we create , for each training image I , a set of up to n image-level structural concept graphs ( I-SCGs ) . Each I-SCG is computed from both the training image I and one of the n c-SCGs : Given the input image I , we use multi-resolution segmentation SLIC ( Achanta et al. , 2012 ) to break the image into patches , which become the concept candidates ( similar as Fig . 4 step 1 ) . In the concept matching step ( similar as Fig . 4 step 2 ) , for each class of interest c , we match features of the segmented patches to the stored anchor representation ( i.e. , mean feature vectors ) of top k concepts deemed important for c using a similarity metric ( e.g . Euclidean distance ) . When at least one of the top k concepts of class c is detected in image I , an I-SCG for class c will be constructed based on the template from the c-SCG of class c. Here I-SCG uses patch features instead of the concept anchor features as node features and calculate the edge features based on spatial relationship between detected concepts in image I . This way we can have up to n I-SCGs generated for the input image I considering all n classes of interest ( More details in Appendix Sec . B.2 ) . GRN takes the I-SCGs as input and we use knowledge distillation to transfer the decision-making logics of the original NN to GRN ( similar to Appendix Fig . 8 ( b ) ( Ge et al. , 2021 ) ) . Using SCG to effectively reveal the reasoning logic of original NN has been validated with extensive experiments in ( Ge et al. , 2021 ) , in which the authors evaluated the logical consistency and faithfulness between SCG explanation and original NN . ( More discussion in Appendix Sec . A.2.2 ) .

Summary: This paper extended VRX (Ge et al. 2021) with an interface to allow human modify the class-specific structural concept graphs (c-SCG) as well as a procedure to distill the human changes in the c-SCGs back to original task's neural networks. This is an interesting piloting idea. However, the scale of the experiments is not convincing enough to demonstrate the generalizability of the performance to a large number of concepts, classes, datasets and complex scenarios.