page_title: Existential risk from artificial general intelligence
section_title: Instrumental goal convergence
hierarchical_section_title: Existential risk from artificial general intelligence / Sources of risk / Instrumental goal convergence
caption_attribution_description: English: Steven Pinker
context_page_description: Existential risk from artificial general intelligence is the hypothesis that substantial progress in artificial general intelligence could someday result in human extinction or some other unrecoverable global catastrophe. It is argued that the human species currently dominates other species because the human brain has some distinctive capabilities that other animals lack. If AI surpasses humanity in general intelligence and becomes "superintelligent", then it could become difficult or impossible for humans to control. Just as the fate of the mountain gorilla depends on human goodwill, so might the fate of humanity depend on the actions of a future machine superintelligence.
  The likelihood of this type of scenario is widely debated, and hinges in part on differing scenarios for future progress in computer science. Once the exclusive domain of science fiction, concerns about superintelligence started to become mainstream in the 2010s, and were popularized by public figures such as Stephen Hawking, Bill Gates, and Elon Musk.
context_section_description: There are some goals that almost any artificial intelligence might rationally pursue, like acquiring additional resources or self-preservation. This could prove problematic because it might put an artificial intelligence in direct competition with humans.
  Citing Steve Omohundro's work on the idea of instrumental convergence and "basic AI drives", Stuart Russell and Peter Norvig write that "even if you only want your program to play chess or prove theorems, if you give it the capability to learn and alter itself, you need safeguards." Highly capable and autonomous planning systems require additional checks because of their potential to generate plans that treat humans adversarially, as competitors for limited resources. Building in safeguards will not be easy; one can certainly say in English, "we want you to design this power plant in a reasonable, common-sense way, and not build in any dangerous covert subsystems", but it is not currently clear how one would actually rigorously specify this goal in machine code.
  In dissent, evolutionary psychologist Steven Pinker argues that "AI dystopias project a parochial alpha-male psychology onto the concept of intelligence. They assume that superhumanly intelligent robots would develop goals like deposing their masters or taking over the world"; perhaps instead "artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no desire to annihilate innocents or dominate the civilization." Russell and fellow computer scientist Yann LeCun disagree with one another whether superintelligent robots would have such AI drives; LeCun states that "Humans have all kinds of drives that make them do bad things to each other, like the self-preservation instinct ... Those drives are programmed into our brain but there is absolutely no reason to build robots that have the same kind of drives", while Russell argues that a sufficiently advanced machine "will have self-preservation even if you don't program it in ... if you say, 'Fetch the coffee', it can't fetch the coffee if it's dead. So if you give it any goal whatsoever, it has a reason to preserve its own existence to achieve that goal."
caption: AI risk skeptic Steven Pinker
exif: {"Image Make": "Canon", "Image Model": "Canon EOS 5D Mark II", "Image Orientation": "Horizontal (normal)", "Image XResolution": "300", "Image YResolution": "300", "Image ResolutionUnit": "Pixels/Inch", "Image Software": "Adobe Photoshop CS5 Windows", "Image DateTime": "2011:03:31 21:08:20", "Image ExifOffset": "216", "Thumbnail Compression": "JPEG (old-style)", "Thumbnail XResolution": "72", "Thumbnail YResolution": "72", "Thumbnail ResolutionUnit": "Pixels/Inch", "Thumbnail JPEGInterchangeFormat": "886", "Thumbnail JPEGInterchangeFormatLength": "4369", "EXIF ExposureTime": "1/640", "EXIF FNumber": "6/5", "EXIF ExposureProgram": "Aperture Priority", "EXIF ISOSpeedRatings": "200", "EXIF SensitivityType": "Unknown", "EXIF ExifVersion": "0221", "EXIF DateTimeOriginal": "2011:03:27 14:22:32", "EXIF DateTimeDigitized": "2011:03:27 14:22:32", "EXIF ShutterSpeedValue": "1165241/125000", "EXIF ApertureValue": "526069/1000000", "EXIF ExposureBiasValue": "0", "EXIF MaxApertureValue": "5/8", "EXIF SubjectDistance": "189/100", "EXIF MeteringMode": "Pattern", "EXIF Flash": "Flash did not fire, compulsory flash mode", "EXIF FocalLength": "85", "EXIF SubSecTime": "40", "EXIF SubSecTimeOriginal": "40", "EXIF SubSecTimeDigitized": "40", "EXIF ColorSpace": "sRGB", "EXIF ExifImageWidth": "2700", "EXIF ExifImageLength": "3600", "EXIF FocalPlaneXResolution": "5616000/1459", "EXIF FocalPlaneYResolution": "1872000/479", "EXIF FocalPlaneResolutionUnit": "2", "EXIF CustomRendered": "Normal", "EXIF ExposureMode": "Auto Exposure", "EXIF WhiteBalance": "Auto", "EXIF SceneCaptureType": "Standard", "EXIF BodySerialNumber": "220107750", "EXIF LensSpecification": "[85, 85, 0/0, 0/0]", "EXIF LensModel": "EF85mm f/1.2L II USM"}