AI gone rogue — Business and reputational risks of building AI products
By: Jacques Verre
Tags: Artificial Intelligence, Risk, Risk Management, Machine Learning

AI gone rogue — Business and reputational risks of building AI products

AI has been migrating over the last couple of years from experimentation and proof of concepts to revenue-generating applications. With over 45% of large companies and 29% of small and medium-sized businesses having adopted AI [1], AI will soon be a core part of many companies.

While the benefits of building AI products is undeniable, it is not without risks that, if not properly managed, can end up being very costly.

With more and more companies now using AI as a revenue generating opportunity, it is important to highlight the risks involved with AI products and the consequences of them going rogue.

What does “going rogue” mean ?

A good way to think about AI is as an automated decisioning system. Data is fed to the system and a prediction or decision is made based on those inputs. Building an AI model simply means making the automated decisions as accurate as possible given the information available at that time.

An AI system can be qualified as having gone rogue when its behavior is not what you expect.

The behavior of AI systems is usually quantified in terms of how accurate they are at a given task. While it can be tedious it is important to understand the impact of a drop in accuracy of 1% in terms of revenue: How much revenue will you leave on the table ?

Drops in accuracy and therefore revenue generated are either:

Gradual : The concept you are modeling has changed over time

: The concept you are modeling has changed over time Abrupt: The AI system does not expect the data you are sending

Gradual drops in accuracy are to be expected for every AI system. This is because the world isn’t static and the behavior you are modeling will change over time. These gradual changes are called concept drift [2] and is the reason why AI systems need to be constantly refreshed.

Abrupt drops in accuracy in AI systems are nearly always the consequence of bugs introduced in the data collection process. As AI systems become more complex, they start depending on data collected within the entire organization. As a result, breaking changes are to be expected in the same way as complex software is never bug free.

As a result of both gradual and abrupt changes in the behavior, all AI systems are accompanied with business and reputational risks.

Risks

Business risks

Both gradual and abrupt changes in behavior result in drop in performance accompanied by lost revenue.

Let’s take the case of an automated bidding system that determines the optimal bid to make. Overtime, that system will stop making optimal decisions and will start overbidding and with each bid your organization will start loosing money. In the case of abrupt changes, it is possible that the AI system with start overbidding massively resulting in big losses or underbidding resulting in a drop in sales due to ads not being shown anymore.

Reputational risks

Having an AI system go rogue can have a big reputational impact for your organization.

A lot of effort and transparency is needed to get end users to start trusting the AI system as by default they tend to not be trusted. A failure to detect and fix issues performance issues will result in a loss of confidence from end users that is very hard to gain back.

The worst way to become aware of an issue in an AI system’s decisions is by having your clients or end-users tell you about it. It is key to have monitoring in place so that you become aware of issues as they arise and are able to fix them quickly.

Mitigating the risks

Given the growth opportunity associated with AI powered products, organizations need to manage the risks rather than simply dismissing AI systems altogether.

In order to mitigate the risks, you should:

Build a model inventory

Implement a monitoring solution

Building a model inventory sounds trivial but without it it is impossible to measure an organization’s exposure. A good model inventory should include a non-technical description, high-level risk assessment and known limitations. While this inventory is maintained by the data team, it should be shared with and reviewed periodically by the senior management team. To learn more about risk management, check out A Framework for managing Model Risk.

In the same way as tools are implemented to monitor software, it is important to have realtime monitoring tools for AI systems. For these tools to be practical, they need to have low latency (time between an issue occurring and it being detected and alerted upon) and allow for configurable alerts. Having an effective monitoring strategy are key to detecting issues before clients notice and avoid incurring both reputational damage and revenue loss.