Merging with AI: How to Make a Brain-Computer Interface to Communicate with Google using Keras and OpenBCI | by Jag Singh | Towards Data Science

Elon Musk and Neuralink want to build a Brain-Computer Interface that can act as the third layer of the brain, allowing humans to form a symbiotic relationship with Artificial Intelligence.
But what if you can already do that?
In a (very) limited form, you actually can.
Brain-Computer Interface (BCI) broadly refers to any system that establishes a direct connection between the nervous system and an electronic device. These devices may be surgically implanted in the brain, or they may be external. Typical paradigms include allowing a user to control an actuator or keyboard, allowing a device to send sensory data to the user, or bilateral communication involving both sensory data and motor control (i.e. a prosthetic arm that receives motor control input and sends sensory data on pressure or temperature)
Historically, neuroprosthetics have been the primary motive for BCI research. These include artificial limbs for amputees, cochlear implants for the deaf, and deep brain stimulation for individuals suffering from seizures. Already, these devices have improved the lives of millions of people, and their widespread use demonstrates the benefits of achieving direct bilateral communication between the brain and electronic devices. However, the possible applications of the technology extend far beyond healthcare. Even within the realm of neuroprosthetics, we can imagine going beyond just repairing and consider augmenting our abilities beyond normal human levels. Artificial limbs may one day progress to the point where they are, by any objective criterion, superior to their natural counterparts. These limbs may look and feel just like normal limbs, but would be far stronger and more agile. Another example would be artificial eyes that are capable of far higher resolution than human eyes, an ability zoom in or out, and to see in the UV or IR spectrum.
The possibilities get even more interesting when considering cognition and skill formation. A recent study demonstrates that stimulating certain parts of the brain improves memory formation and recall. Other experiments have managed to artificially implant memories in animals. As an example, it may be possible to apply the methods of these studies to improve your ability to quickly learn an instrument. Or perhaps it may be possible to combine various neurostimulators and sensors to develop an “arithmetic processing unit” that can detect when particular areas of the brain associated with mathematical or logical reasoning are activated, and communicates with them to enhance abilities.
It is an extension of this cognitive augmentation that Elon Musk and Neuralink want to pursue. According to Musk and many leading AI theorists, a key barrier in humanity’s intellectual progress relative to AI is the bandwidth problem: although computers and AI are becoming ever faster and more capable of processing and generating knowledge, we face immediate and fundamental limitations in our ability to do the same. We acquire information primarily through our senses and ability to interpret language. In the time it takes your eyes and visual cortex to read and understand a single sentence, a computer can scan through thousands of pages of text. It’s conceivable that in a few decades time, we may have advanced AI running on specialized neuromorphic hardware with incredibly accurate models of how the world works and an ability to analyze and understand millions of documents in minutes, making decisions and inferences that are far beyond human comprehension. In a world increasingly dependent on AI driven decision making, humans may find themselves obsolete in all parts of the business, scientific, and political decision making process. Our brains did not evolve to play a game of chess with trillions of pieces or to comprehend calculated strategies that plan millions of moves ahead. It is a fear of this super-intelligent black box that motivates much of the current work at Neuralink, Kernel, and several other related organizations.
Most of the leading edge research in BCI technology seeks to maximize the information bandwidth, typically through invasive methods that implant electrodes directly into the brain or nerves. However, non invasive methods, specifically electroencephalography (EEG) and electromyography (EMG) are routinely used with considerable success. These involve placing electrodes on the surface of your head (EEG) or skin above muscles (EMG) to measure the cumulative electrical activity underneath. The granularity of this data is low, and it is a far cry from the level of precision and bandwidth that will ultimately be needed to realize the more ambitious goals of BCI research. Nevertheless, EEG/EMG enabled BCIs have achieved incredible feats, like controlling drones, video games, and keyboards with thought, and they provide a small glimpse into the possibilities that further research may unlock. Furthermore, several companies like Cognixion and Neurable are exploring real world applications of EEG based BCIs and have received considerable funding and support with many exciting projects underway.
In this project, we establish a direct connection between your nervous system and an external AI agent. This agent may be anything you can get an API for: Google Assistant, Siri, Alexa, Watson, etc. Services like Dictionary or YouTube would qualify as well, but these would limit applications to content queries rather than general purpose requests.
For the purposes of this project, we will query Google Search directly as it provides the most flexibility and is the easiest to set up. Upon completion, you should be able to query a handful of terms on Google simply by thinking about them.
The technique we use exploits the nerve signals generated by your brain in the process of subvocalization. This is the “internal monologue” that takes place inside your head when you slowly and deliberately read or think. You may have noticed yourself doing this when reading silently, sometimes to the point where you’re subtly moving your jaw and tongue without even realizing it. You may also have come across the concept when receiving tips for SAT, MCAT, GRE or other standardized test preparation. Test takers are advised to avoid subvocalization as it’s a bad habit that slows down reading speed.
We are able to exploit subvocalization because the brain sends signals to your larynx corresponding to words you think to say, even if you don’t intend to actually say them out loud. By placing electrodes on your face over the laryngeal and mandibular nerves, we can record signals corresponding to specific words and use them to train deep learning models that discern between different words. In other words (no pun intended), we can discern when you’re thinking about a particular word simply from the act of you thinking about it.
This technology has its limitations, and it is by no means perfect or ready for practical use. However, since its first real world demonstration two years ago by the MIT Media Lab, it has been used successfully in devices that allow users to do math, make phone calls, order pizza, and even receive assistance while playing chess.
The primary hardware tool required is an OpenBCI Ganglion board. There are a variety of other hardware options available, but I found OpenBCI to have one of the largest communities of developers for support. It sets you back about $200, but it’s well worth it given the incredible stuff you can build with it.
In addition to the board, you’ll need electrodes and wires. A set of gold cup electrodes and electrode gel should cost $50 and should work fine.
Ganglion Board
Electrodes
Electrode Gel
Alternatively, you can get a complete OpenBCI starter kit, which includes the board and multiple types of dry electrodes, as well as an electrode headband, for $465. It’s a bit pricey, so the gold cup setup is completely fine. Though, if you plan to experiment with other applications of BCI, like VR (tutorial with Unity VR coming soon!), the headband and dry electrodes make for a far better experience.
Biosensing Starter Kit
OpenBCI also offers 8 and 16 channel boards. These will offer superior data quality, but the 4 channel Ganglion will be adequate for this project.
On a Linux machine, check if you have Python 3.4 or higher. Open your terminal and type the following command:
python3 --version
If you don’t have Python, or if you have an older version, enter:
$ sudo apt-get update$ sudo apt-get install python3.6
Now, download or clone the pyOpenBCI directory.
Change directory into the repository, and run the following command to install the prerequisite packages:
$ pip install numpy pyserial bitstring xmltodict requests bluepy
You’re now ready to install pyOpenBCI
$ pip install pyOpenBCI
To see some action, change directory to pyOpenBCI/Examples and find print_raw_example.py. Open this file with your favorite code editor, and make the following change on line 7:
board = OpenBCICyton(daisy = False)
Should be changed to:
board = OpenBCIGanglion(mac=’*’)
This allows pyOpenBCI to employ the appropriate modules for the particular board we are using.
Now, power on your board.
On your computer, from the Examples directory in your terminal, type the following command:
$ sudo python print_raw_example
Boom!! Your terminal should now be flooded with a stream of raw input data from the board.
Now that we can obtain raw signals, we can start designing and building the data pipeline. To start, we must first convert the raw data into an LSL stream. LSL refers to Lab Streaming Layer, and is a protocol developed at the Swartz Center for Computational Neuroscience at UC San Diego to facilitate the recording and analysis of live data streams. LSL will stream our EEG data onto the local host, from where it can be picked up by other applications or scripts.
Modify the lsl_example.py file in pyOpenBCI/Examples to remove the AUX stream, which we do not need, an add a markers stream:
We must now define an experimental setup that will record the data in the form we want and store it for further use. We want the experiment to generate a data set of time series EEG data separated into intervals, with each interval corresponding to the subvocalization of a single word. To achieve this, we can execute an experiment that starts a recording session of N intervals, with each interval lasting T seconds. All samples within a given interval are annotated with the interval index and the particular word the user is instructed to subvocalize.
The lsl-record.py file from neurotech-berkeley serves as a good starting point. Modify the file in accordance with our defined setup:
You may adjust the termBank (line 64) to try various combinations of words in various contexts. You may also adjust the default duration (line 12) before each session.
Now it’s time for the fun part! Plug the electrodes into your board:
Tape them to your face in the following configuration:
Find a quiet place to sit, and enter the following lines into separate terminals:
// Terminal 1: converts raw data to LSL and streams it$ sudo python lsl_example// Terminal 2: reads LSL data stream and executes experiment$ sudo python lsl_record
Note: we run as sudo to allow the script to detect the MAC address of the board
This should start a recording session of the specified duration. You will be prompted with a random word from your term-bank to subvocalize over 2 second intervals. The recording sessions may get uncomfortable and sleep inducing, so it’s better to do multiple small sessions with breaks in between. Additionally, our experimental setup may lead to poor data quality if frequent disturbances occur (i.e. abrupt movements or subvocalizing the incorrect word).
You may design and implement a more flexible setup with an option to hit a key that deletes the current and previous intervals upon noticing a disturbance. Another workaround is to do multiple small sessions and combine the data at the end, discarding sessions with excessive disturbances. Some noise is inevitable, and you don’t have to be too picky as the model becomes more resilient with increased sample count.
For optimal results, you should have at least 1000 high quality samples for every word in your word bank.
Once you have enough data, it’s time to prepare it for use in machine learning.
Combine and preprocess your data as appropriate so that it has the following format:
Words are indices from 1 to NumIntervals, which is the sum of SessionDuration/2 over the total number of sessions
Terms correspond to the word displayed at each interval
[A, B, C, D] are the EEG channels
Each word, term combination corresponds to approximately 800 lines of data
Import your CSV files into python using numpy. You should have all your data loaded in a NumLines x 6 ndarray in your script.
The first step is to filter the data to remove noise that is outside the frequencies we’re interested in. Informative EEG frequencies correspond to the following bands:
Filtering for frequencies between 4 Hz and 100 Hz may seem reasonable, but would fail because 60 Hz is the frequency of the power grid (may vary by country), which is bound to be a significant source of noise. For optimal results, we should filter between 4 Hz and 50 Hz.
We can use Scipy’s Butterworth filter to select the frequency range we want to keep. Define a filter with the following code:
Then, generate a timestamp column (as we combined multiple datasets and rendered the original timestamps invalid), and apply the filter to each channel:
Once filtered, use the following code to restructure your data into a three dimensional ndarrray array with dimensions IntervalLength x ChannelCount x IntervalCount.
What we’ve effectively done with the above code is converted time series data into image data. It might sound a bit weird, but you can think of each 2 second interval as an image, with each pixel corresponding to the signal value acquired at a particular (channelNumber, lineNumber) coordinate. In other words, we have a stack of IntervalCount images that are each IntervalLength x CannelCount in size.
This technique, demonstrated by Justin Alvey in a similar project, is incredibly powerful because it allows us to treat time series data as if it were image data, allowing us to leverage the power of computer vision and Convolutional Neural Networks (CNNs). You can even visualize a particular subvocalization by plotting it as an image
Additionally, using CNNs allows us to skip Fourier transformations as various frequencies (emerging as patterns on each image) may be learned by the neural network without explicitly specifying what frequencies it should look for.
Now we’re ready to start building the CNN. Since we only have 1 color dimension, we can use a 1D CNN with input dimensions of IntervalLength and ChannelCount. You may experiment with different hyperparameters and architectures. I settled on a single convolution layer, two fully connected layers, and two pooling layers.
For a more detailed analysis on single dimensional CNNs and how they are applied to time series data, refer to this article by Nils Ackermann.
We now have a model that should be able to match an interval of EEG data to a specific word in your word bank.
Let’s see how well it does. Apply the model to the test data, and compare the predicted results against the actual results.
# Test Modely_predicted = model.predict(X_test)
With two words in the term bank, I was able to achieve 90% accuracy. As expected, accuracy diminished slightly with additional words, with 86% for three way and 81% for four way.
One possible way to increase the size of the term bank without compromising accuracy is to create a hierarchical “term tree” with multiple-word queries. You may then perform depth first search on the tree -with each layer of words only being compared to others in that same layer of the same subtree- to find the best match.
We now have all the pieces necessary to query Google using your BCI. Define a mapping between specific subvocalizations and queries, and make the appropriate call:
and....
To do live queries as you think of them, modify and import the lsl_record.py script as a module. You may then call it to read the LSL stream in response to user input for a single 2 second interval.
That’s it! You can now search Google without saying or typing a single word.
You can’t do too much with a three or four word term-bank (barring implementation of the term-tree mentioned earlier). Going through all these steps to search for directions to your nearest gas station is slightly more complicated than just Googling it the normal way. Nevertheless, it’s important to consider where further developments in this technology could lead to. We can imagine an improved and less conspicuous version of this device, not too different from the one the MIT team already has, being used for navigation, web queries, text messaging, smart home management, or any number of routine tasks. When combined with the power of ever improving AI assistants capable of context dependent interpretation, the possibilities expand even further.
The applications of EEG based BCIs are a small subset of what could ultimately be made possible by the cutting edge research taking place at companies and university labs across the world. Telepathic communication, superhuman intelligence, additional senses, simulated experiences, digitization of human consciousness, and merging with artificial intelligence are all worth considering. If these possibilities are realized, they won’t just redefine our relationship with technology: they’ll redefine what it means to be human.
REFERENCES
The following are a list of resources and organizations I found helpful in completing this project and in learning about BCIs in general. I’d like to especially acknowledge the AlterEgo Team at the MIT Media Lab for being the original inspiration for this project, as well as Mr. Alvey and NeuroTech Berkeley for their prior code and tutorial contributions to the BCI community. Additionally, I’d like to thank faculty at the University of California, Davis, notably Dr. Ilias Tagkopoulos, Dr. Karen Moxon, and Dr. Erkin Seker, for ongoing assistance and support.
AlterEgo: A Personalized Wearable Silent Speech Interface
Using deep learning to “read your thoughts” — with Keras and EEG — Justin Alvey
Neurotech Berkeley Github
Open BCI Gitub
Brain-Computer Interfacing: An Introduction — Rajesh Rao
Finally, I’d like to give a huge shout out to the growing BCI/ Neurotech communities which have provided endless support, resources, and enthusiasm for the future.
NeuroTechX
Reddit BCI
Reddit Neuralink
OpenBCI
If you’d like to discuss further or connect, please feel free to reach out on LinkedIn