In particular, the CNN network can combine hierarchical feature extraction and classification together, which plays a role as an automated feature extractor and classifier at the same time. In the field of acoustic signal processing, CNN-based models are also adopted in speech recognition systems [13] for large-scale speech datasets, which shows a great improvement in performance [14] as opposed to more traditional classifiers. More recently, some research [15; 16] has demonstrated that a basic CNN could generally outperform existing methods for environmental sound classification, provided sufficient data. Moreover, the applicability of basic CNN models is also being explored for the bio-acoustic task of whale call detection, such as with respect to North Atlantic right whale calls [17] and Humpback whale calls [18]. The recent successful applications of CNN-based models to time series classification have motivated studies aiming for better input representations of audio signals in order to train the CNN networks more efficiently. Various time-frequency representations, such as spectrograms, can typically offer a rich representation of the temporal and spectral structure of the original audios. A comparative study between different forms of commonly used time-frequency representations is conducted to evaluate their impact on the CNN classification performance of audio data [19]. For superior feature representations for the purposes of time series classification, a Multi-scale time-frequency analysis framework is also proposed to automatically extract features at different scales and frequencies [20]. To thoroughly explore the potential of CNNs on classification of optical remote sensing imagery data, a detailed investigation of the performance of seven well-known CNN architectures is presented in [21; 22], where both two types of training, i.e., fine-tuning of a pre-trained CNN and training of a CNN from scratch, were respectively applied to obtain results. This study aims to apply CNN to efficiently extract the informative features from large datasets of whale calls for classification and similarity analysis. Three main contributions of our work can be summarized as: (1) To overcome the difficulty of lack of sufficient training data and make use of the excellent CNN architectures well trained on a large set of labeled data in computer vision [23; 24], a means of transfer learning was implemented by fine-tuning the trained CNN models, i.e., ResNext101 and Xception [25], on whale-call data to perform the classification task. (2) Both the raw waveforms from the multi-scale sketches of the whale call audios and the log-mel time-frequency representations were explored in order to achieve an ideal feature representation as the input of CNN network. (3) To ensure the convergence of deep neural networks, a Mix-up-based data augmentation technique [26] was also employed, which is advantageous in significantly increasing the number of training data for fully simulating the CNN networks. (4) In addition, the similarity analysis was carried out based on the 'likelihood' output of the Softmax layer, while the whale phylogeny was drawn to further illustrate the detailed relationship among different whale sub-populations. The code can be accessed in the Github ([[https://github.com/Blank-Wang/whale-call-classification](https://github.com/Blank-Wang/whale-call-classification)]([https://github.com/Blank-Wang/whale-call-classification](https://github.com/Blank-Wang/whale-call-classification))).