Build A Voice-Controlled Mouse In 5 minutes | by That Data Bloke | Towards Data Science

In this story, we will build an application using Python that will accept voice commands from the user and perform certain GUI based actions using the mouse and keyboard. You can think of it as you own voice-enabled digital assistant. It can play media, open applications, send emails, move around the mouse pointer and a lot more, all triggered by your voice commands. We will accomplish by using the following two python libraries — PyAutoGUI & Speech_Recognition. All you need is a laptop with Python installed and a microphone.
Before getting into details of how the above-mentioned tools are used in this demo, let us see how they fit into the design of our application.
The diagram below is pretty much self-explanatory. The user inputs the voice commands into the microphone (built-in or external), which is then converted into its equivalent text using the SpeechRecognition module. That converted text is then mapped to certain GUI actions (mouse/keyboard events) performed using the PyAutoGUI module. I have drawn the diagram below for ease.
Let us look at these two libraries in a bit more detail!
The Speech Recognition feature has various applications in the fields of home automation systems, language learning, digital assistants etc. Using this library, we can convert our speech into texts. It supports several APIs (both online & offline). In our example, we will be using the online google API. You might notice some lag at times during the speech to text conversion process, however, among the other API supported by this module, I have found the Google API to be most accurate in my case. It can be downloaded using the following pip commands. The pyaudio is required for using the microphone.
pip install SpeechRecognitionpip install pyaudio
In the below snippet, we are initializing the recognizer object, listening to the input voice from microphone and converting it to text using the “recognize_google” function. We are doing all of that in only the lines marked in Bold letters. The call to the function recognizer.adjust_for_ambient_noise() is optional. This function listens to the input sound and changes the energy_threshold value based on the amount of noise present in the room. It can be thought of as a measure of sensitivity of the recognizer. This value depends on the the ambient noise and higher values generally mean less sensitive. The range of this parameter might be anywhere from 50–4000.
import speech_recognitionrecognizer = speech_recognition.Recognizer()with speech_recognition.Microphone() as src:    try:        audio = recognizer.adjust_for_ambient_noise(src)        print("Threshold Value After calibration:" + str(recognizer.energy_threshold))        print("Please speak:")        audio = recognizer.listen(src)        speech_to_txt = recognizer.recognize_google(audio).lower()        print(speech_to_txt)    except Exception as ex:        print("Sorry. Could not understand.")
Say, instead of microphone, your input is an audio file (file.wav), the line number 3 can be replaced as below:
with speech_recognition.WavFile("file.wav") as src:
Python offers a library called PyAutoGUI for GUI Automation that simulates mouse clicks and keystrokes as if a human user was performing them. For instance, simulating the mouse movements, keystrokes, taking screenshots, maximizing & minimizing a window, there is a lot one can do. You can refer the official documentation link for the complete list. For the purposes of this demonstration, I am using functions like mouse movement, mouse clicks, keyboard presses & find image on screen.
pip install PyAutoGUI
Mouse Controls: To move the mouse cursor across the screen, we need the (x,y) coordinates. Think of the monitor as a 2-D plane (as shown here) with x,y coordinates, where the top left corner is the (0,0). To move to the right, the value of x increases and to go towards the bottom the value of y increases. The pyautogui.size() function returns the dimensions of the screen.
You can check your screen dimension as shown below:
>>> import pyautogui>>> pyautogui.size()Size(width=1920, height=1080)
The below command moves the mouse pointer from its current position to (100,100) position and it takes 0.25 seconds to do so. The next command simulates double click with an interval of 0.25 seconds between the two clicks.
pyautogui.moveRel(100, 100, duration=0.25)pyautogui.click(button='left', clicks=2, interval=0.25)
Keyboard Controls: To simulate key presses, the following function is used. It presses the multimedia ‘mute’ button. The complete list of all the supported keys can be found here.
pyautogui.typewrite(['volumemute'])
Locate Function: There can be times when we do not already know the (x,y) coordinates of the point we want to click (say a button). But if we have a picture of the button stored as an image file, pyautogui.locateOnScreen() function can look for that image pixel by pixel and return the coordinates. One thing to note about the locateOnScreen() feature is that, even if a single pixel does not match, it will fail to detect the image and return a None object. We can then, simply move our mouse to the given location and perform a click. For eg:- I have the Chrome application pinned to my taskbar in my windows machine. I have taken a screenshot (Chrome.PNG) of the icon as below:
Now, the locateOnScreen() function is used like below
icon_location = pyautogui.locateOnScreen(r'screenshots\Chrome.PNG')print(icon_location)>>> Box(left=446, top=1023, width=74, height=52)#Using the left & top values as x & y to click at that locationpyautogui.click(x=446, y=1023, duration=0.25)
That’s all we need to know. Now, the only thing remaining is to map the text to a GUI action. Here’s the mapping I have chosen.
I have attached a demo video of the final application and the code is uploaded here for reference.
In case you happen to read my previous article on Gesture Recognition using neural networks, I had intended to add PyAutoGUI functionality as a complementary part to take actions based on the predicted gestures based on the same principles discussed under the PyAutoGUI section here.
In this demo, we have seen how speech can be converted to text using the Speech Recognition library and how to automate the GUI using PyAutoGUI. By combining these two functionalities, we built a basic version of a voice controlled mouse and keyboard. I hope you find this information useful and use it in your own way.
towardsdatascience.com
medium.com
References:
[1] PyAutoGUI CheatSheet, https://pyautogui.readthedocs.io/en/latest/quickstart.html
[2] Speech Recognition Python, https://pypi.org/project/SpeechRecognition/