A computer-implemented method of identifying abnormal images in a set of medical images for optimal assessment of the medical images, the method comprising: extracting a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features, wherein the global features are assigned the pretrained weights using a first Convolutional Neural Network; extracting a plurality of local features from each medical image by analysing a predefined number of image patches corresponding to each medical image, wherein the predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image into the predefined number of image patches, wherein the respective image patches are analysed to extract local features from the respective image patches, and the local features for the respective image patches are combined to obtain the plurality of local features, wherein a resolution of the medical image is less than the resolution of the higher resolution image; determining an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features, wherein the abnormality score is determined using a pretrained feature classifier; and identifying the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score, wherein the first Convolutional Neural Network is different from a second Convolutional Neural Network used to extract the local features. extracting a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features, wherein the global features are assigned the pretrained weights using a first Convolutional Neural Network; extracting a plurality of local features from each medical image by analysing a predefined number of image patches corresponding to each medical image, wherein the predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image into the predefined number of image patches, wherein the respective image patches are analysed to extract local features from the respective image patches, and the local features for the respective image patches are combined to obtain the plurality of local features, wherein a resolution of the medical image is less than the resolution of the higher resolution image; determining an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features, wherein the abnormality score is determined using a pretrained feature classifier; and identifying the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score, wherein the first Convolutional Neural Network is different from a second Convolutional Neural Network used to extract the local features.