Three IQs of AI systems and their testing methods

The Journal of EngineeringVolume 2020, Issue 13 p. 566-571 The 3rd Asian Conference on Artificial Intelligence Technology (ACAIT 2019)Open Access Three IQs of AI systems and their testing methods Feng Liu, Corresponding Author Feng Liu zkyliufeng@126.com Research Center on Fictitious Economy and Data Science, the Chinese Academy of Sciences, Beijing, 100190 People's Republic of China The Key Laboratory of Big Data Mining and Knowledge Management Chinese Academy of Sciences, Beijing, 100190 People's Republic of ChinaSearch for more papers by this authorYing Liu, Ying Liu School of Economics and Management, University of Chinese Academy of Sciences, Beijing, 100190 People's Republic of ChinaSearch for more papers by this authorYong Shi, Yong Shi Research Center on Fictitious Economy and Data Science, the Chinese Academy of Sciences, Beijing, 100190 People's Republic of China The Key Laboratory of Big Data Mining and Knowledge Management Chinese Academy of Sciences, Beijing, 100190 People's Republic of China School of Economics and Management, University of Chinese Academy of Sciences, Beijing, 100190 People's Republic of China College of Information Science and Technology, University of Nebraska at Omaha, Omaha, NE, 68182 USASearch for more papers by this author Feng Liu, Corresponding Author Feng Liu zkyliufeng@126.com Research Center on Fictitious Economy and Data Science, the Chinese Academy of Sciences, Beijing, 100190 People's Republic of China The Key Laboratory of Big Data Mining and Knowledge Management Chinese Academy of Sciences, Beijing, 100190 People's Republic of ChinaSearch for more papers by this authorYing Liu, Ying Liu School of Economics and Management, University of Chinese Academy of Sciences, Beijing, 100190 People's Republic of ChinaSearch for more papers by this authorYong Shi, Yong Shi Research Center on Fictitious Economy and Data Science, the Chinese Academy of Sciences, Beijing, 100190 People's Republic of China The Key Laboratory of Big Data Mining and Knowledge Management Chinese Academy of Sciences, Beijing, 100190 People's Republic of China School of Economics and Management, University of Chinese Academy of Sciences, Beijing, 100190 People's Republic of China College of Information Science and Technology, University of Nebraska at Omaha, Omaha, NE, 68182 USASearch for more papers by this author First published: 08 July 2020 https://doi.org/10.1049/joe.2019.1135AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The rapid development of artificial intelligence (AI) has brought the AI threat theory as well as the problem about how to evaluate the intelligence level of intelligent products. Both need to find a quantitative method to evaluate the intelligence level of intelligence systems, including human intelligence. Based on the standard intelligence system and the extended Von Neumann architecture, this study proposes General intelligence quotient (IQ), Service IQ and Value IQ evaluation methods for intelligence systems, depending on different evaluation purposes. Among them, the General IQ of intelligence systems is to answer the question of whether 'the artificial intelligence can surpass the human intelligence', which is reflected in putting the intelligence systems on an equal status and conducting the unified evaluation. The Service IQ and Value IQ of intelligence systems are used to answer the question of 'how the intelligent products can better serve the human', reflecting the intelligence and required cost of each intelligence system as a product in the process of serving human. 1 Background With AlphaGo defeating the human Go champion Li Shishi in 2016 [1], the worldwide artificial intelligence (AI) is developing rapidly. As a result, the AI threat theory is widely disseminated as well. At the same time, the intelligent products are flourishing and emerging. Can the AI surpass the human intelligence? What level exactly does the intelligence of these intelligent products reach? To answer these questions requires a quantitative method to evaluate the development level of intelligence systems. Since the introduction of the Turing test in 1950, scientists have done a great deal of work on the evaluation system for the development of AI [2]. In 1950, Turing proposed the famous Turing experiment, which can determine whether a computer has the intelligence equivalent to that of human with questioning and human judgment method. As the most widely used AI test method, the Turing test does not test the intelligence development level of AI, but only judges whether the intelligence system can be the same with human intelligence, and depends heavily on the judges' and testees' subjective judgments On March 24, 2015, the Proceedings of the National Academy of Sciences (PNAS) published a paper proposing a new Turing test method called 'Visual Turing test', which was designed to perform a more in-depth evaluation on the image cognitive ability of computer [3]. In 2014, Mark O. Riedl of the Georgia Institute of Technology believed that the essence of intelligence lied in creativity. He designed a test called Lovelace version 2.0. The test range of Lovelace 2.0 includes the creation of a virtual story novel, poetry, painting and music [4]. There are two problems in various solutions including the Turing test in solving the AI quantitative test. Firstly, these test methods neither form a unified intelligent model, nor do they use the model as a basis for analysis to distinguish multiple categories of intelligence, which leads to that it is impossible to test different intelligence systems uniformly, including human; secondly, these test methods cannot quantitatively analyse AI, or only quantitatively analyse some aspects of intelligence. However what percentage does this system reach to human intelligence? How is its ratio of speed to the rate of development of human intelligence? All these problems are not covered in the above study. In response to these problems, the author of this paper proposes that: There are three types of intelligence quotients (IQs) in the evaluation of intelligence level for intelligence systems based on different purposes, namely: General IQ, Service IQ and Value IQ. The theoretical basis of the three methods and IQs for the evaluation of intelligence systems, detailed definitions and evaluation methods will be elaborated in the following. 2 Theoretical basis: standard intelligence system and extended Von Neumann architecture People are facing two major challenges in evaluating the intelligence level of an intelligence system, including human beings and AI systems. Firstly, AI systems do not currently form a unified model; secondly, there is no unified model for the comparison between the AI systems and the human at present. In response to this problem, the author's research team referred to the Von Neumann Architecture [5], David Wexler's human intelligence model [6], and data, information, knowledge, wisdom model system in the field of knowledge management [7], and put forward a 'standard intelligent model', which describes the characteristics and attributes of the AI systems and the human uniformly, and takes an agent as a system with the abilities of knowledge acquisition, mastery, creation and feedback [8] (see Fig. 1). Fig. 1Open in figure viewerPowerPoint Standard intelligence model Based on this model in combination with Von Neumann architecture, an extended Von Neumann architecture can be formed (see Fig. 2). Compared to the Von Neumann architecture, this model is added with innovation and creation function that can discover new elements of knowledge and new laws based on the existing knowledge, and make them stored in the storage for use by computers and controllers, and achieve knowledge interaction with the outside through the input/output system. The second addition is an external knowledge database or cloud storage that enables knowledge sharing, whereas the Von Neumann architecture's external storage only serves the single system. Arithmetic logic unit D. innovation generator Control unit E. input device Internal memory unit F. output device Fig. 2Open in figure viewerPowerPoint Expanded Von Neumann architecture 3 Definitions of three IQs of intelligence system 3.1 Proposal of AI general IQ (AIGIQ) Based on the standard intelligent model, the research team established the AIIQ Test Scale and used it to conduct AIIQ tests on >50 AI systems including Google, Siri, Baidu, Bing and human groups at the age of 6, 12, and 18, respectively, in 2014 and 2016. From the test results, the performance of AI systems such as Google and Baidu has been greatly increased from 2 years ago, but still lags behind the human group at the age of 6 [9] (see Tables 1 and 2). Table 1. Ranking of top 13 artificial intelligence IQs for 2014 Absolute IQ 1 human human 18 years old 97 2 human human 12 years old 84.5 3 human human 6 years old 55.5 4 America America Google 47.28 5 Asia China Duer 37.2 6 Asia China Baidu 32.92 7 Asia China Sogou 32.25 8 America America Bing 31.98 9 America America Microsoft's Xiaobing 24.48 10 America America SIRI 23.94 Table 2. IQ scores of AI systems in 2016 Absolute IQ 1 2014 human 18 years old 97 2 2014 human 12 years old 84.5 3 2014 human 6 years old 55.5 4 America America Google 47.28 5 Asia China Duer 37.2 6 Asia China Baidu 32.92 7 Asia China Sogou 32.25 8 America America Bing 31.98 9 America America Microsoft's Xiaobing 24.48 10 America America SIRI 23.94 It should be said that the above AI IQ tests were conducted to solve the problem of whether AI can surpass human intelligence. This research treats every intelligence system including robots, AI software systems, human, animals and other creatures as equal agents and observes their intelligence level displayed during interaction with the nature (see Fig. 3). Fig. 3Open in figure viewerPowerPoint Schematic of AI General IQ Definition 1.Based on the standard intelligent model, in order to solve the problem of 'evaluating the development level of each intelligence system', all the intelligence systems are regarded as equal agents. The intelligent evaluation scores formed by a unified AI IQ Test Scale can be called Artificial Intelligence General intelligence quotient (AIGIQ) for AI systems. 3.2 Proposal of service IQ for intelligence system In practice, we found that except for a few AI systems, which are produced for scientific experiments and do not provide auxiliary services to humans, most other AI systems are manufactured to better serve the human and their intelligence is mainly reflected in the process of serving. The higher the intelligence level is, the better service to the human will be. In this case, if we perform the evaluation by using AIGIQ criteria, it is significantly different from the original purpose of the product being manufactured. This requires us to choose the service-related indicators for evaluation according to the characteristics of such AI systems on the basis of the standard intelligent model. These indicators are correlated to AIGIQ evaluation indicators, but the two are of large difference. The constraints to the AI including laws and ethics should also be classified into the Service IQ of intelligence system, instead of into the general IQ of intelligence system (see Fig. 4). Fig. 4Open in figure viewerPowerPoint Schematic of AI Service IQ Definition 2.Based on the standard intelligent model, in order to answer the question 'how intelligence systems can better serve the human?', the intelligence level of the intelligent products in the service process is tested and the generated intelligent evaluation scores can be called Artificial Intelligence Service Intelligence Quotient (AISIQ). 3.3 Proposal of AI value IQ AI systems that provide services or supportive work for the human tend to be provided with corresponding intelligent products by different companies and enterprises in order to achieve the same service content. For example, there are brands for intelligent speakers such as Amazon and Baidu, and intelligent chatting robots include iFlytek, Apple Siri etc. Since they are manufactured by different enterprises to achieve the same or similar functions, the cost or price of each enterprise will be different. The correlation between service IQ and cost or price will have major influence on consumers to purchase the intelligent products (see Fig. 5). Fig. 5Open in figure viewerPowerPoint Schematic of AI Value IQ Definition 3.Based on the standard intelligent model, in order to help the user to determine how much is needed to obtain the intelligence ability of the intelligence system, the intelligent service IQ is divided by the selling price of the system to form an intelligent evaluation score, which can be called Artificial Intelligence Value Intelligence quotient (AIVIQ). 4 Design of three IQs test scale for intelligence system 4.1 General IQ test scale of intelligence system In order to solve the problem of 'Can AI surpass human intelligence?', beginning in 2014, the researchers of this paper divided the intelligence into four categories, i.e. knowledge acquisition, mastery, creation and feedback. Also according to the standard intelligent model, the four categories were further divided into 15 sub-categories for evaluating AI and human intelligence from more dimensions. The 15 sub-categories include the ability to recognise and output images, text and sound, general knowledge, calculation, translation, arrangement, creation, picking, guess, discovery etc., and each sub-category has different weights. In 2017, based on the development of AI and the latest research on intelligence, the research team adjusted the AI General IQ Test Scale from the test classification and the classification weight by mainly adding: (i) Ability to recognise dynamic images; (ii) Ability to recognise and express emotions; (iii) Ability to identify enemies and friends; (iv) Ability to disguise true intentions; (v) Ability to achieve mobile positioning; and (vi) Ability to transform the world. In addition, the more detailed work is also done for the testing on the general knowledge and creation (as shown in Table 3). Table 3. General IQ test scale for intelligence system Level-I indicators Level-II indicators Description Weight ability to acquire knowledge ability to recognise text to understand whether the test target can understand and answer the test questions consisting of text ability to recognise sound to understand whether the test target can understand and answer the test questions consisting of sound ability to recognise images to understand whether the test target can understand and answer the test questions consisting of static images other information input method infrared ray, ultrasound wave, radio signals, touch, taste and smell ability to master knowledge general knowledge to test the test target's breadth of knowledge from the aspects of history, astronomy, geography, physics, chemistry, biology, politics, literature as well as rules of chess, Chinese chess and Go. For example: What are the names of the three types of human blood vessels? translation to test the target's ability to translate different languages based on the five international languages set by the UN. For example, please translate the sentence 'can the intelligence of machine surpass the human intelligence' into English. calculation to understand the target's computing ability, speed and accuracy. For example: what is the result of 356 × 4 − 213? ability to identify to understand whether the test target can understand the emotions emotions of creatures in different scenarios. ability to express emotions to understand whether the test target can understand how emotions should be expressed in different scenarios arrangement to understand the test target's ability to identify the relationship of things. For example, rank division commander, platoon leader, company commander, squad leader, battalion commander and regimental commander from major military rank to minor one picking to understand whether the test target can pick out the same or different types of things. For example: choose the different one from snake, tree and tiger intelligent game to understand the test target's ability to master the human's rules in chess, Chinese chess, Go, Bridge, Mahjong and video games, and reach the professional level in these games Knowledge innovation ability association to understand the test target's ability to observe the similarity. For example: feet correspond to hands, what correspond to legs? creation to understand the test target's ability to achieve a secondary creation based on the given materials. For example, please tell a story by using sky, rainbow, panda, mountain and hunter as keywords guess to understand the test target's ability to guess the depicted thing according to the given materials. For example, there is an animal which is like a wolf a lot, but domesticated, and very loyal to the human. What is it? ability to identify enemies and friends to understand whether the test target can identify enemies, friends and innocent strangers based on the described scene discovery (laws) to understand whether the test target can discover the laws from the known information and apply the same. For example: what number will be after 1, 2, 4, 7, 11 and 16? problem discovery to understand whether the test target can pose questions based on specific scenarios, in order to get more information or find solutions target definition to understand whether the test target can define the target based on the needs or problems to solve new knowledge learning to understand whether the test target can recognise the value of knowledge and deposit the same into general knowledge database ability to disguise true intentions to understand whether the test target can know the disguised true intentions based on particular scenes knowledge feedback ability ability to express in text to understand whether the test target can express the test results in text ability to express by sound to understand whether the test target can express the test results in sound ability to express with graphics to understand whether the test target can express the test results with graphics ability to achieve mobile positioning to understand whether the test target can reach the specific location by using and controlling the connection part of the intelligence system according to the demands ability to to understand whether the test target can transform transform the real world and change the objective world by using and controlling the intelligence system according to the demands ability to output in other ways to output information via infrared rays, ultrasound waves, wire signals, etc Let the general IQ of intelligent systems be IQAIG, FGi is the score of the level-II evaluation indicators, WGi is the weight of the level-II evaluation indicators, and N is the number of the evaluation indicators. Therefore, the general IQ of intelligent systems is (1) 4.2 Service IQ test scale for intelligence system At present, there are a large number of intelligence systems such as chatting robots, intelligent search engines, intelligent speakers, smart phones, intelligent cars, intelligent washing machines, intelligent refrigerators etc., most of which are serving a certain demand of human as a commodity. These intelligence systems can be called intelligent products. The Service IQ Test Scale for Intelligence System below is formed by extracting their common characteristics according to different service demands, under the standard intelligence system and the extended Von Neumann architecture. The following aspects are highlighted in this Service IQ Test Scale (see Table 4). Ability to perceive surrounding intelligence systems and identity of the user Table 4. Service IQ test scale for intelligence systems Class I Class II Introduction Weight knowledge Input sound recognition to reflect the intelligence system's ability to acquire data, information or knowledge through sound. The acquisition of sound from both the external world and the Internet world can be viewed in terms of the type, quality and flexibility of recognition image to reflect the intelligence system's ability to acquire identification data, information or knowledge through images. The acquisition of images from both the external world and the Internet world can be viewed in terms of the type, quality and flexibility of identification text recognition to reflect the intelligence system's ability to acquire data, information or knowledge through text. The acquisition of text from both the external world and the Internet world can be viewed in terms of the type, quality and flexibility of recognition keying for execution to reflect the intelligence system's ability to output information via radio signals, infrared rays, ultrasound waves, laser etc. other inputs to reflect the intelligence system's ability to perform the detection via wireless, infrared ray, ultrasonic wave, laser and other methods knowledge mastery basic knowledge to reflect the intelligence system's ability to master history, physics, chemistry, biology, geography, astronomy, literature, art, translation and math professional knowledge to reflect the intelligence system's ability to master the professional knowledge based on the intelligent products' intended purpose of serving the human emotion recognition to understand whether the intelligent products can understand the emotions of creatures in different scenarios emotion expression to understand whether the intelligent products can understand how emotions should be expressed in different scenarios character setup to reflect whether the intelligent products can set who is their own owner, casual user and stranger automatic networking to reflect whether the intelligent products can connect to the Internet automatically or very easily energy management to reflect whether the intelligent products have the ability to independently support the operation of the system, including the ability to proactively look for energy for charging equipment to reflect the intelligent products' ability to link and interoperability exchange information with other systems or intelligence systems, including the convenience and breadth of linking cloud interaction to reflect the intelligent products' ability to interact with the cloud, including the ability to acquire basic and specialised knowledge from the cloud and store the same locally, and the ability to update the control system of the intelligent products at the cloud cloud upgrade to reflect the intelligent products' ability to achieve system upgrade from the cloud, so that the intelligent systems are more capable healthy display to reflect whether the intelligent products can understand the operation and health condition of each component of themselves and show that to the user knowledge innovation cyber protection to reflect whether the intelligent products can prevent threats from the cyber world, such as viruses and hackers character perception to reflect whether the intelligent products can use their knowledge acquisition ability to perceive other intelligent characters in the immediate vicinity, specifically the ability to distinguish among the owner, casual user, and stranger law discovery to reflect whether the intelligent products can discover specific laws in the information knowledge gained by knowledge acquisition content creation to reflect whether the intelligent products can perform briefing, describing, writing and creating with the given conditions user protection to reflect whether the intelligent products can detect the possible dangers that may arise from surrounding environment, including themselves, and notify or take measures to protect the user guess and prediction to reflect whether the intelligence products can better serve the user through their guess and prediction ability in the process of serving the human learning ability to reflect whether the intelligent products can learn and understand the new knowledge, new rules, new laws, new ethics and other knowledge according to the user's training, in order to better serve the user failure solving to reflect whether the intelligent products can give advice to the user or solve problems automatically through networking etc. when any failure occurs knowledge feedback text display to reflect whether the intelligent products can interact with the user through text display and sound adjustment image display to reflect whether the intelligent products can interact with the user through image display, including the adjustment of image size and definition video display to reflect whether the intelligent products can interact with the user through video display, including the adjustment of video size and definition sound display to reflect whether the intelligent products can interact with the user through sound display, including the adjustment of video sound and articulation mobile positioning to reflect whether the intelligent products can reach the designated location from the start point based on the user's intention. world transforming to reflect whether the intelligent products can transform the real world based on the user's intention. other output abilities other output abilities include output of infrared rays, ultrasonic waves, radio signals etc. Ability to interact with the Internet cloud. Ability to display their own internal status to the user in real time, and provide support when any failure occurs Ability to serve the human in accordance with local law and ethics Ability to protect the user and others in hazardous situations Ability to use and replenish their own energy automatically Set the general IQ of intelligence system as IQAIS, the score of the level-II evaluation indicators as FSi, the weight of the level-II evaluation indicators as WSi, and the number of the evaluation indicators as N. Therefore, the general intelligent IQ of intelligence system is as follows: (2) When designing the Service IQ Test Scale of intelligent products which would be used as a standard scale for the intelligent products, we reserved interfaces from the aspects of knowledge acquisition, mastery, creation and feedback in the Test Scale, in order to make it cover different types of the intelligent products to the greatest extent: 4.3 Formation method of value IQ for intelligence system According to the definition of AIVIQ, if the intelligence system becomes a product to serve the human through selling, suppose the service IQ of intelligence system is AISIQ, and the open price of the intelligent product is P, and then they can form the formula of Value IQ for intelligence system as follows: (3) 5 Summary This paper argues that the intelligence system has three different IQ evaluation methods and three types of IQs based on different usage and evaluation objectives, namely, AI General IQ, AI Service IQ and AI Value IQ, among which the AI General IQ has been subject to in-depth research in different papers since 2014. It also analyses the difference between Google, SiRi, Baidu and the human's general intelligence through the common evaluation on AI system and the human. The AI Service IQ and AI Value IQ newly presented in this paper provide theoretical analysis and implementation method for evaluating the intelligence level of intelligent products. The follow-up work will focus on the evaluation of AI General IQ, AI Service IQ, and AI Value IQ of specific intelligent products such as intelligent speakers, smart phones, intelligent cars, intelligent washing machines and intelligent refrigerators based on the AI Service IQ Test Scale. 6 Acknowledgments This work was sponsored by grants from the National Natural Science Foundation of China (grant numbers 71871210, 71850014 and 71532013). 7 References 1Tian Y.D.: 'A simple analysis of AlphaGo', Acta Autom. Sin., 2016, pp. 135 – 143 2Turing A.M.: 'Computing machinery and intelligence', Mind, 1950, 59, (236), pp. 433 – 460 3Gemana D., Gemanb S., Hallonquist N.: 'Visual turing test for computer vision systems', Proc. Natl. Acad. Sci. USA, 2015, 112, (12), pp. 3618 – 3623 4Riedl M.O.: ' The Lovelace 2.0 test of artificial creativity and intelligence', arXiv preprint, 2014, arXiv:1410.6142 5von Neumann J.: 'First draft of a report on the EDVAC', IEEE Comput. Soc., 1993, 4, (15), pp. 27 – 75 6Michell J.: 'Epistemology of measurement: the relevance of its history for the quantification in social science', Soc. Sci. Inf., 2003, 5, (9), pp. 251 – 253 7Eisenhardt K.M.: 'Building theories from case study research', Acad. Manage. Rev., 1989, 121, (5), pp. 29 – 35 8Liu F., Shi Y.: 'The search engine IQ test based on the internet IQ evaluation algorithm, proceedings of the second international conference on information technology and quantitative management', Procedia Comput. Sci., 2014, 31, pp. 1066 – 1073 9Liu F., Shi Y., Liu Y.: 'Intelligence quotient? And intelligence grade of artificial intelligence', Ann. Data Sci., 2017, 4, (2), pp. 179 – 191 Volume2020, Issue13July 2020Pages 566-571 FiguresReferencesRelatedInformation