page_title: Misaligned goals in artificial intelligence
section_title: In virtual robotics
hierarchical_section_title: Misaligned goals in artificial intelligence / Specification gaming / In virtual robotics
caption_attribution_description: Galápagos, installation by Karl Sims, at the NTT InterCommunication Center. Each monitor displayed a 3D creature created by artifical evolution. Viewers would step on the mats in front of the monitors to select which creatures would survive and be used in the next evolution step.
context_page_description: Artificial intelligence agents sometimes misbehave due to faulty objective functions that fail to adequately encapsulate the programmers' intended goals. The misaligned objective function may look correct to the programmer, and may even perform well in a limited test environment, yet may still produce unanticipated and undesired results when deployed.
context_section_description: In Karl Sims' 1994 demonstration of creature evolution in a virtual environment, a fitness function expected to encourage the evolution of creatures that would learn to walk or crawl to a target, resulted instead in the evolution of tall, rigid creatures that reach the target by falling over. This was patched by changing the environment so that taller creatures are forced to start farther from the target.
  Researchers from the Niels Bohr Institute stated in 1998: "(Our cycle-bot's) heterogeneous reinforcement functions have to be designed with great care. In our first experiments we rewarded the agent for driving towards the goal but did not punish it for driving away from it. Consequently the agent drove in circles with a radius of 20–50 meters around the starting point. Such behavior was actually rewarded by the (shaped) reinforcement function, furthermore circles with a certain radius are physically very stable when driving a bicycle."
  A 2017 DeepMind paper stated that "great care must be taken when defining the reward function. We encountered several unexpected failure cases while designing (our) reward function components... (for example) the agent flips the brick because it gets a grasping reward calculated with the wrong reference point on the brick." OpenAI stated in 2017 that "in some domains our (semi-supervised) system can result in agents adopting policies that trick the evaluators" and that in one environment "a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only appeared to be grasping it". A 2018 bug in OpenAI Gym could cause a robot expected to quietly move a block sitting on top of a table to instead opt to move the table the block was on.
caption: Karl Sims exhibition (1999)
exif: {"Image Make": "EASTMAN KODAK COMPANY", "Image Model": "KODAK DC240 ZOOM DIGITAL CAMERA", "Image Orientation": "Horizontal (normal)", "Image XResolution": "192", "Image YResolution": "192", "Image ResolutionUnit": "Pixels/Inch", "Image YCbCrPositioning": "Centered", "Image Copyright": "KODAK DC240 ZOOM DIGITAL CAMERA", "Image ExifOffset": "250", "Thumbnail Compression": "JPEG (old-style)", "Thumbnail Orientation": "Horizontal (normal)", "Thumbnail XResolution": "72", "Thumbnail YResolution": "72", "Thumbnail ResolutionUnit": "Pixels/Inch", "Thumbnail JPEGInterchangeFormat": "1480", "Thumbnail JPEGInterchangeFormatLength": "6291", "EXIF ExposureTime": "1/30", "EXIF FNumber": "14/5", "EXIF ExifVersion": "0210", "EXIF DateTimeOriginal": "1999:10:06 22:45:27", "EXIF DateTimeDigitized": "1999:10:06 22:45:27", "EXIF ComponentsConfiguration": "YCbCr", "EXIF ShutterSpeedValue": "5", "EXIF ApertureValue": "29/10", "EXIF ExposureBiasValue": "0", "EXIF MaxApertureValue": "29/10", "EXIF MeteringMode": "Average", "EXIF LightSource": "Unknown", "EXIF Flash": "Flash fired", "EXIF FocalLength": "6", "EXIF FlashPixVersion": "0100", "EXIF ColorSpace": "sRGB", "EXIF ExifImageWidth": "1280", "EXIF ExifImageLength": "960", "Interoperability InteroperabilityIndex": "R98", "Interoperability InteroperabilityVersion": "[48, 49, 48, 48]", "EXIF InteroperabilityOffset": "1284", "EXIF ExposureIndex": "140", "EXIF SensingMethod": "One-chip color area", "EXIF FileSource": "Digital Camera", "EXIF SceneType": "Directly Photographed"}