A processor implemented method for classifying heart sound signals, comprising: receiving, via a neural network architecture system, a plurality of heart sound signals specific to one or more users, wherein each heart sound signal from the plurality of heart sound signal is unique and pertains to a corresponding user from the one or more users; splitting the plurality of heart sound signals into a plurality of windows; extracting a set of spectrogram features in frequency domain and a set of Mel Frequency Cepstral Coefficient (MFCC) features in time domain from each of the plurality of windows; concurrently inputting the set of MFCC features to a first Deep Neural Network and the set of spectrogram features to a second Deep Neural Network respectively wherein the input to the first Deep Neural Network are the Mel Frequency Cepstral Coefficients of the heart sound signals for each of the plurality of windows and the input to the second Deep Neural Network are spectrogram images of the heart sound signals for each of the plurality of windows; concurrently learning (i) a temporal behavior by the first Deep Neural Network based on the set of Mel Frequency Cepstral Coefficient (MFCC) features in the time domain and (ii) a spatial behavior by the second Deep Neural Network based on the set of spectrogram features in the frequency domain, wherein the spatial behavior relates to the presence of frequency components; concatenating the learned temporal behavior and the learned spatial behavior to obtain a concatenated behavioral set; wherein during the learning, one or more weights and biases of (i) one or more layers of the first DNN and the second DNN respectively, (ii) a plurality of fully connected layers and a softmax layer of the neural network architecture system are learnt and wherein the output of the first and second Deep Neural Network is concatenated and used as an input for the plurality of fully connected layers; and classifying, using the plurality of fully connected layers and the softmax layer associated with the neural network architecture system, the plurality of heart sound signals as at least one of a normal sound signal or a murmur sound signal based on the concatenated behavioral set. receiving, via a neural network architecture system, a plurality of heart sound signals specific to one or more users, wherein each heart sound signal from the plurality of heart sound signal is unique and pertains to a corresponding user from the one or more users; splitting the plurality of heart sound signals into a plurality of windows; extracting a set of spectrogram features in frequency domain and a set of Mel Frequency Cepstral Coefficient (MFCC) features in time domain from each of the plurality of windows; concurrently inputting the set of MFCC features to a first Deep Neural Network and the set of spectrogram features to a second Deep Neural Network respectively wherein the input to the first Deep Neural Network are the Mel Frequency Cepstral Coefficients of the heart sound signals for each of the plurality of windows and the input to the second Deep Neural Network are spectrogram images of the heart sound signals for each of the plurality of windows; concurrently learning (i) a temporal behavior by the first Deep Neural Network based on the set of Mel Frequency Cepstral Coefficient (MFCC) features in the time domain and (ii) a spatial behavior by the second Deep Neural Network based on the set of spectrogram features in the frequency domain, wherein the spatial behavior relates to the presence of frequency components; concatenating the learned temporal behavior and the learned spatial behavior to obtain a concatenated behavioral set; wherein during the learning, one or more weights and biases of (i) one or more layers of the first DNN and the second DNN respectively, (ii) a plurality of fully connected layers and a softmax layer of the neural network architecture system are learnt and wherein the output of the first and second Deep Neural Network is concatenated and used as an input for the plurality of fully connected layers; and classifying, using the plurality of fully connected layers and the softmax layer associated with the neural network architecture system, the plurality of heart sound signals as at least one of a normal sound signal or a murmur sound signal based on the concatenated behavioral set.