###### Abstract  
Ultrasound (US) imaging is widely tullized as a diagnostic screening method, and deep learning has recently drawn attention for the analysis of US images for the pathological status of tissues. While low image quality and poor reproducibility are the common obstacles in US analysis, the small size of the dataset is a new limitation for deep learning due to lack of generalization. In this work, a convolutional neural network (CNN) using multiple feature maps, such as entropy and phase images, as well as a B-mode image, was proposed to classify breast US images. Although B-mode images contain both anatomical and textual information, traditional CNNs experience difficulties in abstracting features automatically, especially with small datasets. For the proposed CNN framework, two distinct feature maps were obtained from a B-mode image and utilized as new inputs for training the CNN. These feature maps can also be made from the evaluation data and applied to the CNN separately for the final classification decision. The experimental results with 780 breast US images in three categories of benign, malignant, and normal, showed that the proposed CNN framework using multiple feature maps exhibited better performances than the traditional CNN with B-mode only for most deep network models. medical ultrasound; breast US images; deep learning: convolutional neural network: B-mode image; entropy image; phase image +
Footnote †: journal:as a promising machine learning technique, especially in the research domains of natural language processing [3], computer vision [4], and various imaging analyses, including medical images [5; 6]. From the perspective of image analysis, the deep learning approach directly processes the original data and automatically learns multiple levels of abstraction features from images through supervised or unsupervised methods. It has achieved state-of-the-art performance in image analysis with large datasets such as ImageNet [7] in various general tasks, including classification, segmentation, tracking, and object detection. In the areas of medical US image analysis, recent applications of deep learning have involved traditional diagnosis tasks such as tissue classification, tumor detection and segmentation, biometric measurements, and quality assessment. It has also been extended to the clinical domain of US-guided interventions and therapy. However, the main barrier for deep learning in US image analysis is the lack of a well-organized dataset in terms of size and annotated classes in the training set. According to recent papers, most studies utilized a few hundred images to train deep learning networks, which is much smaller than datasets in natural image applications. In addition, for the uniformity of datasets in US applications, most databases were generally acquired from a single-vendor's US imaging system in a single institution. These limitations of US image diversity may degrade the performance of a specific task using deep learning techniques because of the lack of generalization of features and the overfitting of a given dataset. Notwithstanding the above constraints, many studies using deep learning in US image analysis have been conducted and exhibit better performance than traditional machine learning approaches. Breast ultrasound (BUS) is an active application area, since breast cancer is the second leading cause of death for women [8], and early detection and diagnosis is crucial to increase the success of treatment and reduce medical expenses. In 2012, the adaptive deconvolutional network (ADN), which is an unsupervised and generative hierarchical deep model, was applied to classify benign and malignant breast tumors and mass lesions [9]. Along with many other studies of deep learning in US images, [PERSON] et al. proposed a supervised deep learning network, called a deep polynomial network (DPN), for the classification of breast ultrasound images and exhibited the highest performance of 92.4% on small US datasets [10].