Building A Simple Voice Assistant for Your Mac in Python | by Kyle Gallatin | Towards Data Science

Voice assistants like Siri or Cortana can do cool and useful things for you: open applications, search for information, send messages and more. While it’s cool to use the ones you’ve been given...it’s way more fun to make your own. In this post I’ll show you how to start building a voice assistant framework that can do like 1% of things fully developed ones can do on a Mac — sorry but I won’t be covering too much for any other OS.
Like my last blog post on creating a chatbot using your own text messages, this isn’t going to be a fully developed idea and won’t get you too far in a production. Honestly, it took me longer to find/make the gifs for this post than it did to code it out.
This blog post should, however, give you a great place to start if you’ve ever wondered how to create your own voice assistant in python and love to screw around. There’s some other posts out there with pretty much the same general concept — but didn’t see anything much like this on TDS and figured what the hell 🤷🏻 I can write better than that and mine will be more fun.
I’ll show two different voice activated capabilities in this blog post:
Opening applicationsSearching the internet
Opening applications
Searching the internet
We’ll first need 2 special packages. The first one is to get access to the microphone — credit to this post for a great intro to speech recognition in python. The second is the same as my past post, and uses elasticsearch to do query-query matching and return a response.
On Mac, install portaudio and elasticsearch with homebrew. Homebrew is a great package manager for macOS and can be downloaded here. Then in a terminal run:
brew install portaudiobrew install elasticsearch
I’ll be using pip for python package management. The speech_recognition package we’ll use to translate speech to text, and pyaudio is necessary for connecting to your computer’s microphone. We’ll also use the elasticsearch Python client to make things easier for us, I’m working in Python 3.
pip install SpeechRecognitionpip install pyaudiopip install elasticsearch
The speech_recognition package is great because as stated in this aforementioned post — “The SpeechRecognition library acts as a wrapper for several popular speech APIs and is thus extremely flexible. One of these — the Google Web Speech API — supports a default API key that is hard-coded into the SpeechRecognition library.”
Super convenient if you don’t feel like setting up an account for any of the other providers. In less than 10 lines of code, this package should let us connect to our built in computer mic and transcribe our voice!
import speech_recognition as srr = sr.Recognizer()mic = sr.Microphone()with mic as source:    r.adjust_for_ambient_noise(source)    audio = r.listen(source)    transcript = r.recognize_google(audio)    print(transcript)
This code should run until it hears something, then print it out what you said (or what it thinks you said). Super easy. For more on these classes and how that stuff works, refer to aforementioned post.
There’s multiple ways to do this in Python, but the easiest is just to use the one that comes with macOS. Using the subprocess package, we can easily make our computer speak:
import subprocessdef say(text):    subprocess.call(['say', text])say("yo whatup dog")
So now you can transcribe voice to text, make your computer speak and the world is your oyster, sick. Now we can decide how we wanna use the input. For an offhand easy idea my first instinct was to use it to open applications. Obviously it’s not really that cool at all, but it works well as an example, and the general structure I’ll use can be applied to a much wider range of functionality. So let’s first get a list of all our applications.
On Mac, all the apps that I’d want to open are in the /Applications directory and for the most part all have a .app extension, so we can get their names using something like this:
d = '/Applications'apps = list(map(lambda x: x.split('.app')[0], os.listdir(d)))
List the directory and remove the extension. Cool.
Opening the applications is pretty easy too. Here a quick example using the os package and Spotify.
import osapp = 'Spotify'os.system('open ' +d+'/%s.app' %app.replace(' ','\ '))
The .replace is just for properly escaping spaces in longer app names, but if you have Spotify installed and in your applications it should open up.
The obvious solution to tie the voice transcript together with functionality is some kind of exact match, if transcript == 'open spotify' then do this...I probably don’t have to explain how unscalable that is. Besides, it’s way better to get experience with a cool framework like elastic. Also...it wouldn’t be a data science blog post without any actual data or algorithms.
From my last blog post on chatbots — “Elasticsearch is a scalable search platform that uses an algorithm similar to TF-IDF or term frequency inverse document frequency. Essentially, it’s a simple function often used within the search/similarity space that targets documents based on keywords... For a basic overview of it just check wikipedia.”
Basically we’re gonna upload all our voice commands (something like “open spotify”) to elasticsearch and map them to the system commands (the actual programming command) we want to run. To do this, we can quickly create a list of dictionaries.
import osd = '/Applications'records = []apps = os.listdir(d)for app in apps:    record = {}    record['voice_command'] = 'open ' + app.split('.app')[0]    record['sys_command'] = 'open ' + d +'/%s' %app.replace(' ','\ ')    records.append(record)
Now we have all our voice commands and our system commands mapped to one another. Next, open a terminal and start elasticsearch. If you installed it via Homebrew it’ll be in your path so just type elasticsearch and it’ll start on the default port 9200.
Next, we can index our data to make it searchable. We start up the python client and throw our data in a new index named voice_assistant.
from elasticsearch import Elasticsearchfrom elasticsearch.helpers import bulkes = Elasticsearch(['localhost:9200'])bulk(es, records, index='voice_assistant', doc_type='text', raise_on_error=True)
Now your data is all searchable in the index voice_assistant. The response variable we get back from the search function below is generated by the language elastic uses referred to as Query DSL. We’re basically doing a simple TF-IDF search on the voice command with some fuzziness — to account for misspellings/bad transcriptions.
def search_es(query):    res = es.search(index="voice_assistant", doc_type="text", body={                         "query" :{        "match": {            "voice_command": {                "query": query,                "fuzziness": 2            }            }        },    })    return res['hits']['hits'][0]['_source']['sys_command']
Now try it out! search_es('open spotify') should return the command we will pass to os.system which is open /Applications/Spotify.app. Because we have the fuzziness parameter set to 2, search_es('open spofity') will also return the same thing even though there is a misspelling.
Even though this is a small functionality, the key here is that this principle can be applied to a lot of different functionalities — Q&A, other commands, etc...
Now we can transcribe speech to text, and have all our commands properly indexed which we can access with our search_es function. All that’s left it to tie it all together! Let’s create a script that looks for user input, and when it gets it, it searches our index for that command and then tries to run it. The only thing missing is a phrase to trigger our chatbot, so let’s define one more helper function so we have a keyword phrase to activate it, I chose mine to be just “hey dude”.
def activate(phrase='hey dude'):    try:        with mic as source:            r.adjust_for_ambient_noise(source)            audio = r.listen(source)            transcript = r.recognize_google(audio)            if transcript.lower() == phrase:                return True            else:                return False
Using this trigger, we can now more selectively activate our actual search function. Putting it all together with the help of our previously defined functions, we can wrap our code in an infinite loop so it’ll just keep running and waiting for us to activate it. I have my assistant ask me how it can help before running anything, obviously feel free to replace “Kyle” with your name.
import speech_recognition as srimport os from elasticsearch import Elasticsearch##start python client es = Elasticsearch(['localhost:9200'])##initialize speech recognizer and micr = sr.Recognizer()mic = sr.Microphone()while True:       if activate() == True:        try:            say("Hey Kyle, how can I help you today?")            with mic as source:                print("Say Something!")                r.adjust_for_ambient_noise(source)                audio = r.listen(source)                transcript = r.recognize_google(audio)                sys_command = search_es(transcript)                os.system(sys_command)                say("I opened that application for you")        except:            pass    else:        pass
One thing to note — because of all the except statements we’ve got here, you won’t be able to keyboard interrupt out of this once you’ve got it going, you’ll legit have to stop the whole process or if you’re in jupyter restart the kernel — feel free to build in more proper error handling.
Run it and start indiscriminately open applications because you can! It’s cool. Just make sure you’ve started elasticsearch, and defined all your helper functions.
Opening apps is cool but tbh I’m definitely not actually going to use that...having my computer automatically search the internet for whatever I say is a much better use of this assistant. You could probably customize this part more using therequests and BeautifulSoup libraries, but there’s really no point if you can just have selenium open a Chrome/Firefox/whatever window and search for you.
Just use pip to install the necessary python package and make sure you have the webdriver you want to use installed. I’ll use the one for chrome.
pip install selenium
Searching from google is super easy with selenium. I use Chrome as my preferred browser so I start selenium with webdriver.Chrome() below. If you use something else, feel free to replace it.
from selenium import webdriverfrom selenium.webdriver.common.keys import Keysdef search_google(query):    browser = webdriver.Chrome()    browser.get('http://www.google.com')    search = browser.find_element_by_name('q')    search.send_keys(query)    search.send_keys(Keys.RETURN)
That’s it. Selenium opens google in a window, finds the search bar, types for us and then clicks the search button. You can test it real quick to make sure it works:
search_google('how to build a voice assistant in python')
Unlike before we don’t have an elasticsearch index to look into for our information — our searches are more dynamic we have to do some shoddier text preprocessing. Assuming our query will contain the phrase “search google for”, we can just split the string there and pass the rest of the string to our function.
while True:    if activate() == True:        try:            say("Hey Kyle, how can I help you today?")            with mic as source:                print('Say Something!')                r.adjust_for_ambient_noise(source)                audio = r.listen(source)                transcript = r.recognize_google(audio)                phrase = 'search google for '                if phrase in transcript.lower():                    search = transcript.lower().split(phrase)[-1]                    search_google(search)                    say("I got these results for you")                else:                    say('I only search google, sorry fam')        except:            pass    else:        pass
Cool, now you also have a voice assistant constantly listening for you to search the internet or open applications. There’s tons more you can do: actually tie them together in the same assistant (didn’t feel like it) incorporate news/weather APIs, maybe automate email stuff, put your own custom phrases in elasticsearch, build a more complex model for the activation, etc...but this is a decent place to start just for kicks. Hope you enjoyed it!