Eichhoernchen/SiriServer — Issue #45: Handling more than 10 seconds speech
Author: Eichhoernchen
Created: 2012-02-02 18:09:29+00:00
State: open

This is basically just that I don't forget my idea :)

Change speex.py, add reference in constructor to new class that handles speech analysis...
On the StartSpeechRequest an instance of this speech analysis is started in a thread and then waiting for input in a queue (this is a consumer/producer problem),
modify speech.py to not return PCM to calle but rather put each decoded frame into speech analyser directly rather than accumulating until no more available.
On FinishSpeech tell him there is no more data and that it should inform the speech analyser that the end of stream is reached

The speech analyser:
wait for data on queue, process data on queue and only move forward in stream not backwards...
PCM oscillates around zero. Zero means there is no signal, silence... each sample in the PCM is 1/16000s of audio so basically 1/16 ms long.
Todo: find out how long silence periods between words are usually.... make more measurements to see what happens on background noise with the signal during actual silence periods, we probably need a threshold to decide background from foreground.
Count up silent period, ignore minor changes in signal if not too long. if silence periode longer than predefined threshold => end of word found, copy from last_end to current_position - silent_periode_length*0.9 (something along that... lets have some silence after word...)
Continue until silent_periode over, set last_end to current_position - (some milliseconds of silence) and continue from beginning...

With each word do: Create new flac encoder in a thread, encode PCM audio to flac and tell him to send flac packet with a packetnumber (= number of word), start millisec in audio stream, end millisec in stream to HttpClient....

If end_of_stream is reached continue until no more input data is there and then inform HTTPClient about total number of words

HTTPClient must be redone...
also have a queue (again Consumer/producer stuff) for incoming flac packets... allow to make about 3 request to google simultaneous, run each in a thread, max 3 retries... when thread has google answer, finish thread and post packetnumber with google result back to httpClient...
Http Client can reorder packets now by number for correct ordering. If end_of_stream is reached we know total packetnumber and can pass on data to main thread to be processed in process_recognized_speech

process_recognized_speech must be adapted to handle multiple words... adjust response to iDevice to be more like guzzonis answers with confidences for each word... we can also send multiple answers for one word... construct best match by using highest probability of each word in order.