Towards Generic Interface for Human-Neural Network Knowledge Exchange

1 INTRODUCTION . From medicine ( Shen et al. , 2017 ) to self-driving cars ( Buczak & Guven , 2015 ) , machine learning ( ML ) systems are increasingly ubiquitous and outperform humans in multiple tasks . Given the breadth and importance of ML applications , it is critical for humans to use modern machine learning ( ML ) models , such as neural networks ( NN ) , safely and trustfully , and to avoid unpredictable mistakes ( Zerilli et al. , 2019 ) . This requires that humans should be involved in the ML loop . Thus , an interface between human and NN is needed , where NN and human can easily understand , interact , and influence each other . Especially when NN makes mistakes , human prior and domain expertise knowledge with causal inference ability and common sense may help NN achieve better performance . However , the lack of an effective interface makes it hard for a human to locate the reason for NN ’ s error , not to mention correct the error . There are two main challenges for the interaction between humans and NN : ( 1 ) Interpretability , for humans , i.e. , how to understand the reasoning logic of NN and how to correctly locate the reason of errors . ( 2 ) Changing NN ’ s logic and decision , once humans locate the error of NN , i.e. , how to correct it and improve NN ’ s performance ( Fig . 1 ) . To interpret a NN , current pixel-level interpretation methods ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) are limited to low-level relationships . Even recent human-intuitive concept-level explanations ( Ghorbani et al. , 2019a ; Kim et al. , 2018 ) do not reveal the reasoning logic of NN . Visual reasoning explanation ( Ge et al. , 2021 ) mimics the reasoning logic of original NN and provide logical , easy-tounderstand explanations for final decisions , but can not directly influence NN ’ s performance to achieve closed-loop interaction . To interact with NN , current Human-In-the-Loop ML and Interacting ML are ML-centered methods that revolve around a pipeline of re-training a model using human-curated data instead of interacting with their reasoning logic ( Dellermann et al. , 2021 ) . To solve these problems , we propose Human-NN-Interface ( HNI ) for knowledge exchange , with the following key contributions : ( 1 ) HNI use high-level class-specific visual concepts and their relationships to build class-specific structural concepts graph ( c-SCG ) for each class of interest . The c-SCG is a description of the key parts ( concepts ; graph nodes ) of an object class and their spatial relationships ( graph edges ) . HNI allows human and NN to understand each other with c-SCG as a “ language ” to communicate , interact and exchange knowledge . ( 2 ) Through the NN-to-Human path , NN can use c-SCG to show their reasoning logic in a way that is intuitive to human understanding . ( 3 ) Through the Human-to-NN path , a human can analyze the reasoning logic ( c-SCG ) of NN and directly modify it with human prior knowledge in a very shot time ( around 13s to modify each class ) . Then , HNI can use Graph reasoning Network and partial knowledge distillation to transfer knowledge from human back to NN , such that the NN obtains new knowledge from humans . ( 4 ) By creating new c-SCGs or modifying existing c-SCGs , humans can “ teach ” NN about new objects they have never seen before , which provides a new pipeline to achieve zero-shot learning . 2 RELATED WORK . We review research areas that share similarities with our work , to position our contributions . Human-AI Interaction for Machine Learning ( ML ) applications aims to best combine human domain expertise and computational power of ML . To satisfy this need , Human-In-the-Loop ML ( HILML ) ( Dellermann et al. , 2021 ) and Interactive Machine Learning ( IML ) ( Ware et al. , 2002 ) have recently emerged . However , both are ML-centered methods which let humans play a “ server ” role around the ML process , from data production , ML modeling , to model evaluation and refinement ( Maadi et al. , 2021 ) . This limits human involvement and domain expert performance . “ Tell me where to look ” ( Li et al. , 2018 ) uses an explainable attention map to correct segmentation errors of NN , which is thus limited to low-level relationships . User interaction was also introduced in the image generation task , Interactive Image Generation ( Mittal et al. , 2019 ) can repeatedly modify images based on modifications to the scene graph while keeping the contents generated over previous steps . In our work , we take a step toward an interface with which human users and NN can more efficiently communicate , interact and exchange knowledge between each other , with no dependency on a given list of attributes . Interpreting neural networks . Research on interpretability methods for neural networks ( NNs ) has received more and more attention . Some try to explain NN by visualizing the correlation between each pixel of the input image and the final outputs . For instance , CAM ( Zhou et al. , 2016 ) and Grad-CAM ( Selvaraju et al. , 2017 ) can generate class-specific attention maps . Differently , several recent works focus on more human-intuitive concept-level explanations . ACE and TCAV ( Ghorbani et al. , 2019a ; Kim et al. , 2018 ) proposed algorithms to extract meaningful concepts from images and then produce an understandable explanation . VRX ( Ge et al. , 2021 ) uses visual concepts to explain an NN ’ s reasoning logic . Based on this work , our HNI goes beyond this and we treat logic explanation as a foundation , allowing humans to directly modify NN ’ s logic . Then HNI can transfer human ’ s knowledge back to the original NN to achieve a closed-loop interaction between humans and NN . Graph Neural Networks ( GNN ) are deep learning methods that operate on graph domains , which learn to represent graph nodes , edges , or subgraphs by low-dimensional vectors ( Zhou et al. , 2021 ) . Considering the trackable information-communication properties of GNNs , many reasoning tasks have adopted GNNs as a tool , such as VQA ( Teney et al. , 2017 ; Norcliffe-Brown et al. , 2018 ) , scene understanding ( Li et al. , 2017a ) , and semantic explanations ( Ge et al. , 2021 ) . In this work , we use a GNN-based Graph Reasoning Network ( GRN ) as the bridge between NNs and humans . It takes SCG as input and transfers knowledge with NN through knowledge distillation . Knowledge distillation can effectively learn a small student model from a large ensembled teacher model using soft targets ( Hinton et al. , 2015 ) . Knowledge distillation has been widely used in privileged learning ( Lopez-Paz et al. , 2015 ) , adversarial defense ( Papernot et al. , 2016 ) , learning with noisy data ( Li et al. , 2017b ) . In this work , we use knowledge distillation to transfer knowledge between a NN and a GNN-based graph reasoning network ( GRN ) which encodes human ’ s prior knowledge to achieve human-NN knowledge exchange . Zero-shot learning aims to train a classifier that can classify testing instances that belong to classes that are never seen before . Existing works include using the one-vs-rest solution ( Verma & Rai , 2017 ) , synthesizing pseudo instances of the unseen class ( Guo et al. , 2017 ) , projecting feature space instances and semantic space prototypes into a common space ( Palatucci et al. , 2009 ) , and using similar seen class as the positive instance of the unseen class ( Gan et al. , 2016 ) . Huynh & Elhamifar ( 2020 ) conduct compositional zero-shot learning by using a feature composition framework to extract and combine features of attributes to construct fine-grained attributes for unseen classes . Jia et al . ( 2021 ) use active zero-shot learning which promotes human-AI teaming by actively modifying the class-attribute matrix . However such attribute labels may not be always available in real-world scenarios . Here , we propose a new pipeline with HNI , which makes no assumption on the availability of attribute label , humans help build the understanding of the new class using learned visual concepts and structure . Then HNI transfers the knowledge of unseen class back to NN . 3 HUMAN-NN INTERFACE . Our proposed Human-NN Interface ( HNI ) to bridge the interaction between human and neural networks is visually summarized in Fig . 2 . There are two main path . 3.1 NN-TO-HUMAN . NN-to-human path explains the NN ’ s reasoning logic for each decision ( instance-level explanation , see Appendix Sec . A.1 ) and more importantly , the understanding of NN for each class , represented as class-specific Structural Concept Graph ( c-SCG ) . Each c-SCG is bound to one class ( Fig . 2 shows the c-SCG of school bus ) , where the nodes represent the important visual concepts that original NN considered most important in identifying the class of interest , and edges represent the pairwise structural relationships ( dependencies ) between concepts . As shown in Fig . 2 ( top ) , given a trained NN , there are two main steps in order to explain the reasoning logic of NN to human users : ( 1 ) Using Visual Concept Extractor ( VCE ) to discover representative visual concepts for each class of interest . The detailed procedure follows Appendix Fig . 8 ( a ) of ( Ge et al. , 2021 ) : To discover concepts for each class , we collect 50 to 100 images of the class . We first use top-down gradient attention ( Grad-Cam ( Selvaraju et al. , 2017 ) ) to constrain the relevant regions for concept proposals to the foreground segments , thereby ruling out irrelevant background patterns for this class . Then we follow the same workflow as the ACE paper ( Ghorbani et al. , 2019b ) : multi-resolution segmentation , feature extraction , clustering patches in latent space to obtain the concept candidates , and sorting these concept candidates based on importance score , similar to ( Kim et al. , 2018 ) . After that we obtain the concept pool ( each concept is represented by one mean feature vector , sorted by importance score ) for each class , which will serve as a source of concept candidates when human users modify concepts ( nodes ) for c-SCG . To build c-SCG that reveals the reasoning logic for each class of interest of the original NN , we select the top k ( k=4 in our experiments ) important concepts and their mean feature vectors as nodes . Each directed edge in an SCG edgeji = ( vj , vi ) has two attributes : 1 ) representation of the spatial structural relationship between nodes ; 2 ) dependency eji ( a trainable scalar ) between concepts vi , vj . The edge features can reveal the underlying causal logic of interactions between visual concepts which is crucial for the final decision . c-SCG extract such relationship between visual concepts during training and incorporate them as edge features in c-SCG . By showing c-SCG for each class of interest , NN-to-human path provides easy-to-understand insights for human users on the reasoning process of NN , which is also a foundation for the Human-to-NN path . The edges ( structural relationship and dependency ) are fully connected at the beginning and then , using the learning in the following step 2 , we select and only keep the important edges . ( 2 ) Using Graph Reasoning Network ( GRN ) to mimic the decision-making process of original NN with knowledge distillation . As a GNN-based network , GRN takes a graph as input . To train GRN based on the built c-SCGs ( one for each of n classes of interest ) , we need to establish connections between training images and the c-SCGs . To this end , we create , for each training image I , a set of up to n image-level structural concept graphs ( I-SCGs ) . Each I-SCG is computed from both the training image I and one of the n c-SCGs : Given the input image I , we use multi-resolution segmentation SLIC ( Achanta et al. , 2012 ) to break the image into patches , which become the concept candidates ( similar as Fig . 4 step 1 ) . In the concept matching step ( similar as Fig . 4 step 2 ) , for each class of interest c , we match features of the segmented patches to the stored anchor representation ( i.e. , mean feature vectors ) of top k concepts deemed important for c using a similarity metric ( e.g . Euclidean distance ) . When at least one of the top k concepts of class c is detected in image I , an I-SCG for class c will be constructed based on the template from the c-SCG of class c. Here I-SCG uses patch features instead of the concept anchor features as node features and calculate the edge features based on spatial relationship between detected concepts in image I . This way we can have up to n I-SCGs generated for the input image I considering all n classes of interest ( More details in Appendix Sec . B.2 ) . GRN takes the I-SCGs as input and we use knowledge distillation to transfer the decision-making logics of the original NN to GRN ( similar to Appendix Fig . 8 ( b ) ( Ge et al. , 2021 ) ) . Using SCG to effectively reveal the reasoning logic of original NN has been validated with extensive experiments in ( Ge et al. , 2021 ) , in which the authors evaluated the logical consistency and faithfulness between SCG explanation and original NN . ( More discussion in Appendix Sec . A.2.2 ) .

Summary: The paper provides a way for explaining the reasoning of a neural network to humans in the form of a class-specific structural concept graph (c-SCG). The c-SCG can be modified by humans. The modified c-SCG can be incorporated in training a new student model. Experiments show that the new model performs better on classes that their corresponding c-SCG have been modified.