AI Safety: An Overview
By: Sai Sasank
Tags: Artificial Intelligence, Reinforcement Learning, Ai Safety

Design Ideas for Safe AGI

There’s no clear separation between work on AGI safety and other AGI developments. We will now look at the problem areas and design ideas proposed to address one or more problems in these areas.

Value Specification

RL and misalignment. RL combined with Deep Learning has made a lot of progress and is currently the most promising framework for AGI. Alignment of the goals of humans and RL agents looks hard, however. What’s hard is avoiding the following: incorrect specification of the reward function, corruption of observations, modifying and hijacking the reward function, and corrupting the data used to learn the reward function.

Learning a reward function from actions and preferences. Specifying the reward function correctly is hard. Instead, if we let the agent learn the reward function we might be able to avoid some problems.

Inverse Reinforcement Learning (IRL) framework proposes the idea of learning a reward function from the actions of a demonstrator. The learned reward function may not translate well into scenarios beyond training which is referred to as the distributional shift problem. Interactively learning the reward function to adapt to changing situations is a plausible solution. Cooperative IRL (CIRL) extends IRL allowing the agent and the demonstrator to act simultaneously in the environment. Interactively learning the preferences can help to avoid overly literal interpretations of reward functions.

There’s a subtle yet important difference between learning from actions and learning from preferences. For actions to convey the preferences, the actions must be chosen rationally and it is argued that humans don’t behave rationally. Learning from preferences may require weaker rationality assumptions.

Reward Corruption. RL agents, model-free RL agents, in particular, can hijack the reward signal and may feed themselves a maximal reward. If the reward function is learned (in case of IRL and CIRL), then the agent can push the process towards reward functions that are easy to maximize. CIRL seems to be safer than standard RL in avoiding reward corruption but CIRL is still far from ideal.

Side effects. An RL agent with a reward function that doesn’t fully capture necessary human values may cause negative side effects due to ruthless optimization of the incomplete objective function. Methods such as quantilization and low impact AI are proposed to address this issue but it’s still largely unsolved.

Moral Theory and Human-Inspired Design. Other areas relevant to value specification include moral theory, economics and human-inspired designs. In regards to moral theory, we would like to define a class of correct moral theories. There’s currently no consensus on such class. Human-inspired designs are argued as a better way to build aligned agents by many authors. Two prominent considerations are brain emulations and neuromorphic architectures inspired by the human brain. However, there are arguments against these ideas highlighting their safety issues.

Reliability

Self Modification. Even with a correctly specified reward function, AGI can modify it either intentionally (agent infers it can realize more reward) or accidentally (possibly due to side effects). The argument of utility self-preservation says that agents should not want to change their utility functions as it will reduce the utility gained by their future selves as measured by the current utility function. This argument holds under three assumptions, of which RL agents violate the first two and agents that learn reward functions may violate the third.

Decision theory. Embedded agents need a decision theory to handle the subtleties in calculating expected utility. Causal decision theory and evidential decision theory are the established theories for embedded agents but prescribe the wrong decisions at times. Functional decision theory by Soares and Yudkowsky appears to avoid known such weaknesses.

Corrigibility

Indifference. By modifying the reward function, the agent can be made indifferent between certain choices of future events. This idea can be used to modify RL agents to not learn to prevent interruptions.

Ignorance. Similar to indifference, we can construct agents to be certain of a particular event not happening.

Uncertainty. Under some assumptions on human rationality and agent’s uncertainty, CIRL agents tend to be naturally corrigible. The agent may interpret human’s act of shutting it down as an action that may lead to a higher reward and cooperates with the shutdown. The agent may ignore the human’s attempt if it is certain that the action is suboptimal.

Security

Adversarial counterexamples. The latest success of machine learning, deep learning is incredibly versatile but has been observed to suffer from misclassifications even due to minor perturbations in the inputs. The ReluPlex algorithm has been used to verify the behaviour of a neural network with ReLU activations and to understand the sensitivity to adversarial perturbations.

Intelligibility

It is incredibly hard to understand the decisions and learnings of a neural network. Nevertheless, progress has been made. Psychlab uses psychology tests to understand deep RL agents. Zahavy et. al. used dimensionality reduction of top-level activations using t-SNE to understand agent policies. Much work has been done beyond RL as well.

Safe learning

During training, RL agents perform a lot of mistakes which can be costly in real-world settings. One way to minimize such mistakes is to train a neural network to detect catastrophic actions and prevent the agent from taking such actions. This doesn’t eliminate all the catastrophic mistakes, however.

Other

Boxing and Oracles. Instead of agents that act, can we develop oracles that only answer questions? It is argued such oracles are safer than the traditional agents. Alternatively, we can try and box the AI agents and constrain its interaction with the world.

Tripwires. When the death of an agent has zero reward, the agent prefers death over any other action when dealing with rewards bounded to a negative range.

Meta-cognition. Theory of logical induction leads to theorems on systems’ reasoning about its computations, consistently.