A speech processing device comprising: a speech detector adapted to detect speech of individual speakers from acoustic signals; a total-amount-of-speech calculator adapted to calculate total amount of speech for each of all pairs of the speakers in each of individual segments defined by dividing a determination time period, the total amount of speech being the sum of the number of frames in which the speech from one of the two speakers is detected and the number of frames in which the speech of the other is detected; an established-conversation calculator adapted to calculate degrees of established conversation of all the respective pairs of the speakers in each of the segments on the basis of the detected speech; a long-time feature calculator adapted to calculate a long-time feature of the degrees of established conversation within the determination time period for each of the pairs; and a conversational-partner determining unit adapted to extract a conversation group holding conversation from the speakers on the basis of the calculated long-time feature, wherein the degree of established conversation is a value being determined in accordance with a rate of a time when one of the two speakers gives speech and the other gives no speech to the individual segments, the long-time feature is the average of the degrees of established conversation in the determination time period, and the established-conversation calculator invalidates the degree of established conversation in the segment having the total amount of speech smaller than a predetermined threshold in the calculation of the long-time feature. a speech detector adapted to detect speech of individual speakers from acoustic signals; a total-amount-of-speech calculator adapted to calculate total amount of speech for each of all pairs of the speakers in each of individual segments defined by dividing a determination time period, the total amount of speech being the sum of the number of frames in which the speech from one of the two speakers is detected and the number of frames in which the speech of the other is detected; an established-conversation calculator adapted to calculate degrees of established conversation of all the respective pairs of the speakers in each of the segments on the basis of the detected speech; a long-time feature calculator adapted to calculate a long-time feature of the degrees of established conversation within the determination time period for each of the pairs; and a conversational-partner determining unit adapted to extract a conversation group holding conversation from the speakers on the basis of the calculated long-time feature, wherein the degree of established conversation is a value being determined in accordance with a rate of a time when one of the two speakers gives speech and the other gives no speech to the individual segments, the long-time feature is the average of the degrees of established conversation in the determination time period, and the established-conversation calculator invalidates the degree of established conversation in the segment having the total amount of speech smaller than a predetermined threshold in the calculation of the long-time feature.