A processor implemented method, comprising: obtaining, via one or more hardware processors, (i) a speech signal corresponding to one or more users, and (ii) a corresponding text transcription from the speech signal; splitting, via the one or more hardware processors, the speech signal to a plurality of speech segments; determining, via the one or more hardware processors, one of (i) one or more emotion words-based speech segments, or (ii) one or more non-emotion words-based speech segments from the plurality of speech segments based on (a) the speech signal, (b) the obtained text transcription, and (c) a language specific emotion words-based dictionary, wherein the one or more non-emotion words-based speech segments are speech segments without emotion words and the emotion words-based speech segments with emotion words determined from the language specific emotion words-based dictionary, wherein the language specific emotion words-based dictionary is populated in real-time based on relevant words from the obtained text transcription including positive emotion words or negative emotion words; selecting, via the one or more hardware processors the one or more non-emotion words-based speech segments as one or more training segments and exclude the speech segments containing the emotion words; and training, via the one or more hardware processors, one or more classifiers using the one or more selected training segments of the one or more non-emotion words-based speech segments to obtain one or more trained classifiers, wherein the one or more trained classifiers predict emotional valence of a test speech signal comprising speech segments not used as the one or more training segments while training the one or more classifiers, wherein the emotional valence prediction results are compared to a ground truth to calculate the accuracy of the prediction by the one or more trained classifiers, wherein a best classifier amongst the one or more trained classifiers is chosen based on comparing the emotional valence prediction results of individual trained classifiers or a fusion of combination of two or more trained classifiers. obtaining, via one or more hardware processors, (i) a speech signal corresponding to one or more users, and (ii) a corresponding text transcription from the speech signal; splitting, via the one or more hardware processors, the speech signal to a plurality of speech segments; determining, via the one or more hardware processors, one of (i) one or more emotion words-based speech segments, or (ii) one or more non-emotion words-based speech segments from the plurality of speech segments based on (a) the speech signal, (b) the obtained text transcription, and (c) a language specific emotion words-based dictionary, wherein the one or more non-emotion words-based speech segments are speech segments without emotion words and the emotion words-based speech segments with emotion words determined from the language specific emotion words-based dictionary, wherein the language specific emotion words-based dictionary is populated in real-time based on relevant words from the obtained text transcription including positive emotion words or negative emotion words; selecting, via the one or more hardware processors the one or more non-emotion words-based speech segments as one or more training segments and exclude the speech segments containing the emotion words; and training, via the one or more hardware processors, one or more classifiers using the one or more selected training segments of the one or more non-emotion words-based speech segments to obtain one or more trained classifiers, wherein the one or more trained classifiers predict emotional valence of a test speech signal comprising speech segments not used as the one or more training segments while training the one or more classifiers, wherein the emotional valence prediction results are compared to a ground truth to calculate the accuracy of the prediction by the one or more trained classifiers, wherein a best classifier amongst the one or more trained classifiers is chosen based on comparing the emotional valence prediction results of individual trained classifiers or a fusion of combination of two or more trained classifiers.