Patent ID: 11961533
Assignee: THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK
Field: Computer technology (Electrical engineering)
Classification: CPC G  H  A | IPC A  G

Claim 6:
7. A system comprising:
at least one microphone to obtain a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;
one or more neural sensors to obtain neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to; and
a controller coupled to the at least one microphone and the one or more neural sensors, the controller configured to:
generate a combined sound spectrogram from the combined sound signal;
apply speech-separation processing to the combined sound spectrogram to derive multiple resultant speaker spectrograms that each corresponds to a different one of the multiple sound sources; and
select one of the multiple resultant speaker spectrograms based on the obtained neural signals for the person, including to:
generate an attended speaker spectrogram based on the neural signals for the person;
compare the attended speaker spectrogram to the derived multiple resultant speaker spectrograms to select one of the multiple resultant speaker spectrograms; and
transform the selected one of the multiple resultant speaker spectrograms into an acoustic signal;

wherein the controller configured to apply the speech-separation processing to the combined sound spectrogram is configured to apply neural-network-based speech-separation processing to the combined sound spectrogram to derive the multiple resultant speaker spectrograms, including to:
separate the combined sound spectrogram representation of the combined signal into the multiple resultant speaker spectrograms by feeding the sound spectrogram representation of the combined sound spectrogram into multiple deep neural networks (DNNs), each of which is trained to separate the combined sound spectrogram into a specific speaker spectrogram for a specific speaker from the multiple speakers, to obtain the multiple speaker spectrograms.