Repository: alexa-pi/AlexaPi
Issue category: Performance Issue

Issue:
Streaming audio to and from AVS
While waiting for Echo push notifications for a year now, I did a small extension of https://github.com/sammachin/AlexaPi to essentially enable push by submitting pre-recorded questions to AVS when triggered by an event. It worked pretty well, but the problem I faced is that response time is slow, typically 5 or even 10 seconds. I believe the approach used in https://github.com/alexa/alexa-avs-sample-app address those performance issues by streaming the audio to AVS (and maybe also streaming the response back, I asked that question, hoping to hear back from them). The approach is explained here - https://forums.developer.amazon.com/questions/10982/avs-takes-too-much-time-to-respond.html. 

Is streaming audio to and from AVS in the plan for the new evolving AlexaPi?



Hints:
Sounds interesting. We're not there yet to do these _future_ things - we need to refactor the AVS code first, but we ain't that far either. Let's keep it here for when we get there. 

Also, I'm not sure that streaming **from** AVS is gonna have significant performance impact, because Alexa's answers are usually very small filesize-wise. Streaming **to** Alexa is probably gonna have noticeable impact, because from my experience, upload speed is often crap.

Here is a terrific example of implementing streaming to AVS using the v1 API - https://github.com/respeaker/Alexa/blob/master/alexa.py

If anyone implemented this, that would be awesome.
I thought someone was working on the new API ? 
Yep, but we might do this for the old also, since it might not be such a big change (I dunno, just guessing) and the new API integration + testing + setting it as default is gonna take some time.
@alexa-pi/team 

Okay, I got this working (streaming to Amazon) with our AlexaPi - basically just copied a small portion of code from @respeaker and holy shit! If you don't have an Echo / Dot or haven't tried ReSpeaker you will not believe how fast it can get! I'll open a PR today and I'd like to release it this weekend as a new version (and bump all those planned versions by one), because this is definitely worth it. Stay tuned!
Awesome. Count me in if you need someone to test it :-)

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 80fd89a6..7410c952 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/).
 
 ### Changed
 - On Debian-based systems, the `python-pip` package gets uninstalled and `pip` is installed via `easy_install` instead to get the latest version.
+- Recorded audio is now streamed to AVS instead of recording the whole thing and then sending it at once.
+    - Brings huge speed improvement (response latency).
+    - This means that when your recording LED (or whatever your device equivalent) is on, data gets sent to Amazon already.
+    - Used code from @respeaker (thank you!).
 
 ### Fixed
 - Updated old versions of requirements in `requirements.txt`. Also fixes `ImportError: No module named cheroot.server`.
diff --git a/src/alexapi/capture.py b/src/alexapi/capture.py
index edad3e02..57e3c3c1 100644
--- a/src/alexapi/capture.py
+++ b/src/alexapi/capture.py
@@ -4,7 +4,7 @@
 import alsaaudio
 import webrtcvad
 
-from alexapi.exceptions import ConfigurationException
+from .exceptions import ConfigurationException
 
 logger = logging.getLogger(__name__)
 
@@ -55,62 +55,76 @@ def silence_listener(self, throwaway_frames=None, force_record=None):
 		inp.setrate(self.VAD_SAMPLERATE)
 		inp.setformat(alsaaudio.PCM_FORMAT_S16_LE)
 		inp.setperiodsize(self.VAD_PERIOD)
-		audio = b''
 
-		start = time.time()
-
-		do_VAD = True
-		if force_record and not force_record[1]:
-			do_VAD = False
-
-		# Buffer as long as we haven't heard enough silence or the total size is within max size
-		thresholdSilenceMet = False
-		frames = 0
-		numSilenceRuns = 0
-		silenceRun = 0
+		debug = logging.getLogger('alexapi').getEffectiveLevel() == logging.DEBUG
 
 		logger.debug("Start recording")
 
 		if self._state_callback:
 			self._state_callback()
 
-		if do_VAD:
-			# do not count first 10 frames when doing VAD
-			while frames < throwaway_frames:
+		def _listen():
+			start = time.time()
+
+			do_VAD = True
+			if force_record and not force_record[1]:
+				do_VAD = False
+
+			# Buffer as long as we haven't heard enough silence or the total size is within max size
+			thresholdSilenceMet = False
+			frames = 0
+			numSilenceRuns = 0
+			silenceRun = 0
+
+			if debug:
+				audio = b''
+
+			if do_VAD:
+				# do not count first 10 frames when doing VAD
+				while frames < throwaway_frames:
+					length, data = inp.read()
+					frames += 1
+					if length:
+						yield data
+
+						if debug:
+							audio += data
+
+			# now do VAD
+			while (force_record and force_record[0]()) \
+					or (do_VAD and (thresholdSilenceMet is False) and ((time.time() - start) < self.MAX_RECORDING_LENGTH)):
+
 				length, data = inp.read()
-				frames += 1
 				if length:
-					audio += data
+					yield data
 
-		# now do VAD
-		while (force_record and force_record[0]()) \
-				or (do_VAD and (thresholdSilenceMet is False) and ((time.time() - start) < self.MAX_RECORDING_LENGTH)):
+					if debug:
+						audio += data
 
-			length, data = inp.read()
-			if length:
-				audio += data
+					if do_VAD and (length == self.VAD_PERIOD):
+						isSpeech = self._vad.is_speech(data, self.VAD_SAMPLERATE)
 
-				if do_VAD and (length == self.VAD_PERIOD):
-					isSpeech = self._vad.is_speech(data, self.VAD_SAMPLERATE)
+						if not isSpeech:
+							silenceRun += 1
+						else:
+							silenceRun = 0
+							numSilenceRuns += 1
 
-					if not isSpeech:
-						silenceRun += 1
-					else:
-						silenceRun = 0
-						numSilenceRuns += 1
+				if do_VAD:
+					# only count silence runs after the first one
+					# (allow user to speak for total of max recording length if they haven't said anything yet)
+					if (numSilenceRuns != 0) and ((silenceRun * self.VAD_FRAME_MS) > self.VAD_SILENCE_TIMEOUT):
+						thresholdSilenceMet = True
 
-			if do_VAD:
-				# only count silence runs after the first one
-				# (allow user to speak for total of max recording length if they haven't said anything yet)
-				if (numSilenceRuns != 0) and ((silenceRun * self.VAD_FRAME_MS) > self.VAD_SILENCE_TIMEOUT):
-					thresholdSilenceMet = True
+			logger.debug("End recording")
 
-		logger.debug("End recording")
+			inp.close()
 
-		inp.close()
+			if self._state_callback:
+				self._state_callback(False)
 
-		if self._state_callback:
-			self._state_callback(False)
+			if debug:
+				with open(self._tmp_path + 'recording.wav', 'wb') as rf:
+					rf.write(audio)
 
-		with open(self._tmp_path + 'recording.wav', 'wb') as rf:
-			rf.write(audio)
+		return _listen()
diff --git a/src/main.py b/src/main.py
index 7b6ade20..a8c1cd8a 100755
--- a/src/main.py
+++ b/src/main.py
@@ -13,6 +13,7 @@
 import optparse
 import email
 import subprocess
+from future.builtins import bytes
 
 import yaml
 import requests
@@ -277,27 +278,33 @@ def renew(self):
 			logger.critical("AVS token: Failed to obtain a token: " + str(exp))
 
 
-def alexa_speech_recognizer():
-	# https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/rest/speechrecognizer-requests
-	logger.debug("Sending Speech Request...")
-
-	platform.indicate_processing()
+# from https://github.com/respeaker/Alexa/blob/master/alexa.py
+def alexa_speech_recognizer_generate_data(audio, boundary):
+	"""
+	Generate a iterator for chunked transfer-encoding request of Alexa Voice Service
+	Args:
+		audio: raw 16 bit LSB audio data
+		boundary: boundary of multipart content
+	Returns:
+	"""
+	logger.debug('Start sending speech to Alexa Voice Service')
+	chunk = '--%s\r\n' % boundary
+	chunk += (
+		'Content-Disposition: form-data; name="request"\r\n'
+		'Content-Type: application/json; charset=UTF-8\r\n\r\n'
+	)
 
-	url = 'https://access-alexa-na.amazon.com/v1/avs/speechrecognizer/recognize'
-	headers = {'Authorization': 'Bearer %s' % token}
 	data = {
 		"messageHeader": {
-			"deviceContext": [
-				{
-					"name": "playbackState",
-					"namespace": "AudioPlayer",
-					"payload": {
-						"streamId": "",
-						"offsetInMilliseconds": "0",
-						"playerActivity": "IDLE"
-					}
+			"deviceContext": [{
+				"name": "playbackState",
+				"namespace": "AudioPlayer",
+				"payload": {
+					"streamId": "",
+					"offsetInMilliseconds": "0",
+					"playerActivity": "IDLE"
 				}
-			]
+			}]
 		},
 		"messageBody": {
 			"profile": "alexa-close-talk",
@@ -306,12 +313,38 @@ def alexa_speech_recognizer():
 		}
 	}
 
-	with open(tmp_path + 'recording.wav', 'rb') as inf:
-		files = [
-			('file', ('request', json.dumps(data), 'application/json; charset=UTF-8')),
-			('file', ('audio', inf, 'audio/L16; rate=16000; channels=1'))
-		]
-		resp = requests.post(url, headers=headers, files=files)
+	yield bytes(chunk + json.dumps(data) + '\r\n', 'utf8')
+
+	chunk = '--%s\r\n' % boundary
+	chunk += (
+		'Content-Disposition: form-data; name="audio"\r\n'
+		'Content-Type: audio/L16; rate=16000; channels=1\r\n\r\n'
+	)
+
+	yield bytes(chunk, 'utf8')
+
+	for audio_chunk in audio:
+		yield audio_chunk
+
+	yield bytes('--%s--\r\n' % boundary, 'utf8')
+	logger.debug('Finished sending speech to Alexa Voice Service')
+
+	platform.indicate_processing()
+
+
+def alexa_speech_recognizer(audio_stream):
+	# https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/rest/speechrecognizer-requests
+
+	url = 'https://access-alexa-na.amazon.com/v1/avs/speechrecognizer/recognize'
+	boundary = 'this-is-a-boundary'
+	headers = {
+		'Authorization': 'Bearer %s' % token,
+		'Content-Type': 'multipart/form-data; boundary=%s' % boundary,
+		'Transfer-Encoding': 'chunked',
+	}
+
+	data = alexa_speech_recognizer_generate_data(audio_stream, boundary)
+	resp = requests.post(url, headers=headers, data=data)
 
 	platform.indicate_processing(False)
 
@@ -396,7 +429,7 @@ def process_response(response):
 		try:
 			data = bytes("Content-Type: ", 'utf-8') + bytes(response.headers['content-type'], 'utf-8') + bytes('\r\n\r\n', 'utf-8') + response.content
 			msg = email.message_from_bytes(data) # pylint: disable=no-member
-		except TypeError:
+		except AttributeError:
 			data = "Content-Type: " + response.headers['content-type'] + '\r\n\r\n' + response.content
 			msg = email.message_from_string(data)
 
@@ -427,10 +460,10 @@ def process_response(response):
 
 						player.play_speech(resources_path + 'beep.wav')
 						timeout = directive['payload']['timeoutIntervalInMillis'] / 116
-						capture.silence_listener(timeout)
+						audio_stream = capture.silence_listener(timeout)
 
 						# now process the response
-						alexa_speech_recognizer()
+						alexa_speech_recognizer(audio_stream)
 
 				elif directive['namespace'] == 'AudioPlayer':
 					if directive['name'] == 'play':
@@ -481,15 +514,10 @@ def trigger_callback(trigger):
 
 
 def trigger_process(trigger):
-	if event_commands['pre_interaction']:
-		subprocess.Popen(event_commands['pre_interaction'], shell=True, stdout=subprocess.PIPE)
 
 	if player.is_playing():
 		player.stop()
 
-	if trigger.voice_confirm:
-		player.play_speech(resources_path + 'alexayes.mp3')
-
 	# clean up the temp directory
 	if not debug:
 		for some_file in os.listdir(tmp_path):
@@ -500,12 +528,18 @@ def trigger_process(trigger):
 			except Exception as exp: # pylint: disable=broad-except
 				logger.warning(exp)
 
+	if event_commands['pre_interaction']:
+		subprocess.Popen(event_commands['pre_interaction'], shell=True, stdout=subprocess.PIPE)
+
 	force_record = None
 	if trigger.event_type in triggers.types_continuous:
 		force_record = (trigger.continuous_callback, trigger.event_type in triggers.types_vad)
 
-	capture.silence_listener(force_record=force_record)
-	alexa_speech_recognizer()
+	if trigger.voice_confirm:
+		player.play_speech(resources_path + 'alexayes.mp3')
+
+	audio_stream = capture.silence_listener(force_record=force_record)
+	alexa_speech_recognizer(audio_stream)
 
 	triggers.enable()
 
diff --git a/src/requirements.txt b/src/requirements.txt
index c073524d..531394f1 100644
--- a/src/requirements.txt
+++ b/src/requirements.txt
@@ -5,4 +5,5 @@ pyalsaaudio>=0.8.4
 webrtcvad>=2.0.10
 pyyaml
 pocketsphinx>=0.1.3
-coloredlogs
\ No newline at end of file
+coloredlogs
+future>=0.16.0
\ No newline at end of file


Test patch: