AI as Augmented Intelligence
In the upcoming years AI will primarily serve as “augmented intelligence” to assist us in our everyday lives and help us do our jobs…
Category: Artificial Intelligence
Author: Waterway Data

AI as Augmented Intelligence
In the upcoming years AI will primarily serve as “augmented intelligence” to assist us in our everyday lives and help us do our jobs better.
AI has traditionally referred to “artificial intelligence” coming from the imagination of science fiction: omnipotent robots and sentient intelligences in the cloud, changing the world drastically but also stoking our fears of losing control (and our jobs) to our own creations.
AI as Augmented Intelligence
But this is not the reality of what AI is actually doing for us, nor what AI will bring in the coming decade.
Actually AI is going to be much more “Augmented Intelligence” than artificial intelligence. Its primary accomplishment will be to assist us in our everyday lives, and to help us do our jobs better by “augmenting” and multiplying what types of problems we humans can solve on our own.
This term “augmented intelligence” has been promoted by IBM as what AI should be. It was coined in 1962 by Douglas Engelbart:
“Increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems.
…We do not speak of isolated clever tricks that help in particular situations. We refer to a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human feel for a situation usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids.”
-“Augmenting Human Intellect” (DARPA, 1962)
Indeed I think IBM and Engelbart are going to be proven right: AI acting as “Augmented Intelligence” will power the next era of ubiquitous voice-based personal assistants in the next era of consumer technology, the “Assistant Era”, and will begin to gain real traction with businesses by helping employees do their jobs better, faster, cheaper, and smarter.
AI in the new Assistant Era
For decades in the 1980s-1990s, many hoped that full-fledged artificial intelligence was just around the corner. But by the time of the “Web Era” (approx. 1996–2008), the reality was that the algorithms, computing power, available data sets, and hardware were just not ready to deliver talking robots or true AI ready for applications in the real world. We still got Google, which fundamentally changed how we access and consume information.
Then came the “Mobile Era” (approx. 2008–2017) where each app company wanted to automate some aspect of our lives through our new smartphones. As it turned out, we were just not ready to hand over our lives to an app. We didn’t want apps knowing everything about us and eerily suggesting what we should do next, without our asking. If we’re being told by our phone that right now is the time to turn left at the next stoplight / buy this product / save up $100 / watch this TV show / book this hotel … we want to know why we should do so. We need to believe this app is helping us and somehow figuring out what’s really in our own interest, not bombarding us with marketing. Few apps successfully crossed into the realm of doing this intelligently and with our consent.
Then in 2014 Amazon’s Echo and Alexa planted the seed of the new era. They changed the game by pairing together: (1) Speech recognition algorithms and deep learning that that could finally do voice-to-text with an error rate below 5%, plus (2) A high-quality speaker in a form factor that could sit in your kitchen and continuously listen for a hot word. This proved the magic formula to make us start to believe a friendly anthropomorphized being was actually sitting in our kitchen just waiting to set a timer for 10 minutes or deliver the day’s news. (Even my kids tell me not to be “mean” to Alexa when I get a little annoyed with her limited vocabulary.)
iPhone vs. Echo sales since year of launch (source: Forrester Research, Statista/Apple)
Alexa proved the catalyst to drive unexpectedly high sales and jumpstart a new market — just like when the iPhone paired a low-latency touch screen interface with a CPU/RAM/HD that could all fit within a device the size of your hand, creating a new user experience intuitive enough for a billion people to use out of the box.
Fast forward to today (believe it or not, it is 2018). Speech recognition within Google Assistant, Google Voice Keyboard, Apple’s Siri and voice-recognition within iMessage or Apple TV, and Microsoft’s Cortana have caught up and even surpassed Amazon’s Alexa when it comes to high-quality NLP and chat that can understand most of what we say to it — primarily thanks to the success of Deep Learning trained on very large data sets of voice queries.
We are starting to hold basic conversations with the nice personalities embodied in Alexa, Siri, Google, and Cortana with the expectation that they can talk back to us intelligently through our phones, homes, tablets, TVs, and cars. Hundreds of other chatbots built by startups, banks, and eCommerce companies are all getting ready to talk to us too.
The user experience of talking is faster and better than typing on a tiny keyboard or navigating the visual UI of a small screen. The reason is simple: we can speak and understand speech at about 150 words per minute, but the average person can only type at about 40 words per minute on a phone’s keyboard (and with 8% errors).
Thus we have quietly entered the beginning of the “Assistant Era”. There’s no going back to the “Mobile Era” where innovation was driven (and funded) primarily by sales of phones or apps. Apps are now becoming just one of several channels through which we talk to an Assistant or engage with some offline service like Uber.

However — much like the iPhone and Android in the early days brought promise but ultimately very limited functionality — the Assistants of this new era still have a long way to go before they can have an intelligent conversation with us, not just respond to a set of simple queries or commands (and only when we use exactly the right words). Even Facebook saw the shortcomings in their Messenger “M” chatbot and took it down.
To meet the demand that Assistants get better at understanding us and responding intelligently, the AI researchers and engineers will need to keep innovating fast in a few fields:
Natural Language Understanding (NLU): Understanding exactly what we mean by words like “her”, “love”, “firms”, “grow”, “global stocks”, or “there” is surprisingly tricky to teach a computer to do, despite our brains’ ability to naturally figure out meaning within the context of a speaker or situation. NLP/NLU techniques are developing ways to infer semantics through (a) vector-space modeling and autoencoding of patterns in what words are typically used together in a sentence like word2vec and GLoVe, (b) named entity recognition to identify what entities a document’s nouns/pronouns/indexicals point to, (c) intent recognition and topic modeling to infer what a chunk of text is talking about or trying to communicate, and (d) dialog managers to keep track of the flow of a conversation and determine how to best respond in the face of ambiguity (disambiguation), to fill in missing information (slot filling), and to interpret different ways people say the same thing (variations).
Recommender Systems: What do we humans want? It’s hard enough to understand this about ourselves, let alone teach AI to predict it for us. Yet we judge the usefulness of any Assistant or chatbot by the subjective quality of what it suggests to us and the help it offers, not just how well it responds to precise queries. Let’s say a chatbot says, “Would you like help?” on an eCommerce site and you respond, “Yeah, I’m looking for nice dress shoes to go with my jeans.” The chatbot needs to quickly decide what are the right few dress shoe products to suggest (that actually do go well with jeans), otherwise you‘ll quickly determine this is just a dumb, useless bot. The field of recommender systems is solving this by moving beyond collaborative filtering and linear matrix factorization (think: Netflix and Amazon.com) into coupling Deep Learning with recommender systems to model non-linear preferences and create a latent vector-space model of users and products, doing sentiment classification, and ranking suggestions using many diverse features via supervised machine learning algorithms like learning-to-rank.
Machine Translation: Google Translate has employed Deep Learning with very-deep neural networks and vector-space modeling of semantics across languages to automatically translate between any of over 100 languages. Google’s NLP and Translation APIs will let developers incorporate translation into their own Assistants in new, creative ways.
Augmented Intelligence to help us do our jobs better (not take our jobs away)
High user growth will come from Assistants for consumer applications. But high revenue growth for the AI industry will come first from businesses and their employees.
Fearing robots will replace us, literally (source)
The widespread fear of AI’s future in the workplace is this: AI will replace humans and put thousands (or millions) of people out of work. Jobs will shift away from low-wage labor to high-wage/highly-skilled jobs in technology and management, increasing inequality. The trucking industry is supposedly the harbinger, where 3 million jobs in the US will be obviated by self-driving trucks.
Side-stepping the moral/economic/political debate around whether this should be acceptable or unacceptable for society…I believe the fear is unwarranted and a red herring, because that is just not going to happen.
Why? Technology does not leapfrog its way into society against the interests of consumers, employees and employers. Technology takes “the path of least resistance” to gain traction through incremental applications used by real people who are comfortable with that technology and are willing to pay for it.
In business the early adopters need to be employees themselves. Those employees need to be the ones to choose to buy AI-based services, so the path of least resistance for AI will be to start by providing “augmented intelligence” to help us human employees do our jobs better. Replacing us in large numbers is not only infeasible (technically), but going to be too costly for business owners to consider in all but specific industries, for many years.
The future driving experience according to Mercedes-Benz
To illustrate let’s look at the prospect of Self-driving Cars / Autonomous Driving, one of the most exciting and well-hyped uses of AI technology. Google proclaimed as early as 2012 that we would be riding in self-driving cars by 2017. Yet here we are in 2018 and we are probably still at least another decade away from truly autonomous cars being manufactured en masse. Why?
It turns out that achieving Level 5 autonomous driving — where the driver is entirely uninvolved in the driving process — is a lot harder than we were led to believe. It is not enough for the autonomous car to drive around sensing the road and other cars or pedestrians through vision/lidar/radar (relative positioning and object detection). It needs to know where it is on the road in order to slow down in time for an upcoming stoplight or get into the right-most lane to proceed off the exit ramp (absolute positioning). It needs to handle conditions with impeded visibility like snow, darkness, rain, or glare. It needs to predict what other drivers are going to do like in a crowded intersection (multi-agent systems), and even communicate with the other autonomous vehicles (connected vehicles). It needs to make decisions under uncertainty like whether to slow down because there’s a truck crossing the lane or keep going because that’s just a sign overhead (planning, and even ethics). To solve all this together will require significant coordination between not only (a) the autonomous driving software providers/startups and (b) auto manufacturers, but also (c) sensor manufacturers, (d) Tier-1 parts suppliers, (e) road mapping companies, (f) local/state governments and regulatory bodies, and (g) the consumers themselves. That kind of coordination effort between many parties will inevitably take years.
Financially, the total cost of the stack of software+hardware+services to power autonomous driving will need to come down below a few thousand dollars before this would be cheaper on net than the wages paid to truck or delivery drivers. That will all take years to happen — I’d wager at least 10 years — and then it will take another 10 years or so to swap out the existing inventory of cars/trucks already on the road with new autonomous vehicles.
Instead actually the kind of automation being offered so far by Tesla, GM, Ford, Google/Waymo, Uber and startups like Drive.ai, Cruise/GM, Zoox, nuTonomy/Delphi, Nuro.ai, and Embark is Level 2 or Level 3 autonomy, where the car drives itself only in certain circumstances like on highways, on known routes, or while parking, and otherwise relies on the human driver to take back over. The role of this kind of AI is to assist the driver, not replace the driver.
Businesses are beginning to demand AI as augmented intelligence in a few different forms. In Artificial Intelligence for the Real World (Harvard Business Review, Jan-Feb 2018), Thomas Davenport and Rajeev Ronanki surveyed 250 executives and found that:
51% of companies said it was to “enhance the features, functions, and performance of our products”,
36% said it was to “free up workers to be more creative by automating tasks” or to “optimize internal business operations”
only 22% (the least common answer) said it was to “reduce head count through automation.”
They also manually analyzed 152 projects involving AI and found 38% focused on detecting patterns in large datasets using machine learning, while 47% focused on robotic process automation (RPA):
“One might imagine that robotic process automation would quickly put people out of work. But across the 71 RPA projects we reviewed (47% of the total), replacing administrative employees was neither the primary objective nor a common outcome. Only a few projects led to reductions in head count, and in most cases, the tasks in question had already been shifted to outsourced workers.”
To keep up with this demand, the AI industry will need to continue finding ways to help employees accomplish a task more easily and more quickly than they could on their own. Here are some business applications of AI and Machine Learning being implemented right now:
A vector-space representation of word meanings (source)
Text Processing: Well-trodden techniques in natural language processing (NLP), text retrieval, and text mining are being combined in new ways with machine learning to extract insights and knowledge from large bodies of text then deliver simple summaries or suggestions for documents to read. For instance in the legal field, startups like Casetext are combing through millions of legal case decisions to tell attorneys what they should read. Or large companies fielding volumes of customer support inquiries can better triage high-priority tickets, help their customer support agents respond with relevant answers, or monitor what things are being said on social media streams. This is coming primarily from startups applying recent AI techniques developed in academia like autoencoders, learning-to-rank, LDA text clustering, vector space modeling, or hierarchical topic/intent modeling (like Lang.ai).
Anomaly Detection: Deep Learning is getting better at learning how to find the “needle in the haystack” when a single actor is too different from the normal pattern of behavior and should be flagged— such as when detecting insurance claims fraud, credit card/payments fraud, hacking and cybersecurity threats, Medicare fraud, or bank account takeover. This can be done by training a decision tree/random forest/gradient boosting model on training data, or using newer techniques like autoencoders or Bayesian/Gaussian cluster estimation even when prior training data are not available.
Risk Assessment: All underwriting decisions, credit risk assessments and mitigation of default risks can be done across complex data sets by neural network-based classification/regression models or decision trees/random forests/gradient boosting, and faster, cheaper and with higher accuracy than traditionally done manually by Ops teams. Lending startups like Affirm, Upstart, Change, and Underwrite.ai all use machine learning at the core of their underwriting and risk assessments.
Identification: Convolutional neural networks are very good at identifying when something is the same as something else, even when there are many variations/dimensions/features to sift through. For instance in the United States now we are all introduced to face recognition built into the iPhone X — which figures out that the face shown to the iPhone is the same person as the original face the iPhone saw during setup. Expect to see this deployed widely in security cameras for surveillance or “badging in” to get into a building, as is already done in China. The same technology is used to do fingerprint recognition in almost all iPhone/Android phones, and also is being applied in mobile device fingerprinting where the unique characteristics of a single device make it a “key” for identification of a user online.
Predicting Prices: AI is capable of finding complex patterns in time-series data like pricing and using them to forecast future prices with high accuracy. For instance in real estate, machine learning kernel regression and vector-space similarity modeling are used to predict prices of homes not yet on the market, such as in Zillow’s Zestimates, Redfin Estimates, or AirBnb’s suggested prices for rental units, by finding similarities with other homes even in a different part of the country. Similarly, pricing of commodities and investments are being forecasted using ML regression models by new kinds of quant hedge funds.
Lead Qualification: Salesforce is leading the charge in developing AI and ML for the sales process, with a significant investment in their Einstein product line. Sales and marketing professionals so often need to decide whether one customer lead is more likely to convert than another, and AI is capable of doing this better than most humans can. This can be accomplished using ML classification using decision trees/random forest or deep neural networks with logistic regression/softmax output units. Salesforce is funding research to innovate new lead scoring methods, too.
Product Recommendations: Every eCommerce or content site with many products or pieces of media to sell needs to determine which products to target to which users. In the early 2000s, Amazon.com, Netflix, and Pandora innovated by using collaborative filtering, matrix factorization and ML-based recommender systems to do this well. Now startups are integrating more sophisticated vector-space models of products’ similarities and user preferences into their recommendation engines. Also this can be applied in other fields like medicine, where AI can assist doctors in recommending the right diagnosis/treatment to the right patient based on a complex knowledge base.
The time for AI has arrived, and it won’t be just a passing hype cycle. The techniques developed by the AI research community are being integrated more and more into applications for both consumers and business.
Of course AI will not be all beneficial for society —just like with mobile phones and social media, it will take a while to digest and overcome the downsides new technology brings. But in the next decade AI‘s contribution will be primarily to help us. It will augment our intelligence, help us be more productive, more creative, and hopefully live more enriched lives, too.
Have any other predictions for what AI will become in the next few years? Leave your thoughts in the comments.
About Us:
Waterway Data is a consulting firm offering on-demand AI, machine learning, and data science. Please get in touch if your product or business could be made more intelligent using the latest in AI technology.