Review of state-ofthe-arts in artificial intelligence . Present and future of AI

Here, I review current state-of-the-arts in many areas of AI to estimate when it’s reasonable to expect human level AI development. Predictions of prominent AI researchers vary broadly from very pessimistic predictions of Andrew Ng to much more moderate predictions of Geoffrey Hinton and optimistic predictions of Shane Legg, DeepMind cofounder. Given gigantic rate of progress in recent years and this broad range of predictions of AI experts, AI safety questions are also discussed. 1 Natural language processing Let’s begin with some state-of-the-arts in natural language generation. Table 1: Best perplexity scores dataset perplexity link Wikipedia english corpus snapshot 2014/09/17 (1.5B words) 27.1 [1] 1B word benchmark (shuffled sentences) 24.2 [2] OpenSubtitles (923M words) 17 [17] IT Helpdesk Troubleshooting (30M words) 8 [17] Movie Triplets (1M words) 27 [18] PTB (1M words) 62.34 [19] Why perplexity matters? If neural net says smth inconsistent (in any sense: logical, syntactic, pragmatic) then it means that it gives too much probability to some inappropriate words i.e, it’s perplexity isn’t optimized yet. When hierarchical neural chat bots would achieve low enough perplexity, they would likely to write coherent stories, to answer intelligibly with common sense, to reason in a consistent and logical way etc. Just as in [141] we would be able to adjust a conversational model to imitate style and opinions of a distinct person. What are reasonable predictions for perplexity improvements in the nearest future? Let’s take two impressive recent works both submitted to arxiv in February 2016. They are: ”contextual LSTM models for large scale NLP tasks” [1] and ”Exploring the limits of language modeling” [2]. Table 2: Best perplexity scores, single model number of hidden neurons perplexity in [1] perplexity in [2] 256 38 – 512 – 54 1024 27 – 2048 – 44 4096 – – 8192 – 30 24.2 (ensemble) From table 2, it’s quite reasonable to predict that for [1] hs=4096 might give pplx<20, hs=8192 might give pplx<15 and ensemble of models with hs=8192 trained on 10B words might give perplexity well below 10. Nobody can tell now what kind of common sense reasoning would such a neural net have. It’s quite probable that even in this year the discussed decrease in perplexity might allow us to create neural chat bots which write reasonable stories, answer intelligibly with common sense, discuss things. All of this would be just a mere consequence of low enough perplexity. Shannon estimated lower and upper bounds of human perplexity to be 0.6 and 1.3 bits per character [34]. Strictly speaking, applying formula (17) in his work gives lower bound equal 0.648 which he rounded. Given average word length of 4.5 symbols and including spaces (as Shannon included them in his game) gives us an estimate for lower bound on human-level word perplexity as 11.8 = 20.648∗5.5 or better to say 10 plus something. This lower bound might be much less than real human perplexity [30]. Another source of improvement may come from solving some discrepancy in what kind of perplexity is optimized [35] [36]. However, both KL(P||Q) and KL(Q||P) have optimum when P=Q. Also, [35] [36] propose ways to partially solve that problem using adversarial learning. ”Generating sequences from continuous space”[97] demonstrates impressive advantages of adversarial paradigm. Table 3: Best BLEU scores lang pair BLEU dataset link en=>fr 37.5 WMT’14 [3] fr=>en 35.8 WMT’14 [4] ar=>en 56.4 NIST OpenMT’12 [5] ch=>en 40.06 MT03 [6] ge=>en 29.3 WMT’15 [7] en=>ge 26.5 WMT’15 [7] en=>ru 29.37 newstest-14 [8] en=>ja 36.21 WAT’15 [9] Human BLEU score for chinese=>english translation on MT03 dataset is 35.76 [15]. In recent article [6] neural network gets 40.06 BLEU on the same task and dataset. They took state-of-the-art [16] ”GroundHog” network and replaced maximum likelihood estimation with their own MRT criterion, which increased BLEU from 33.2 to 40.06. Here is quote from abstract: ”Unlike conventional maximum likelihood estimation, minimum risk training is capable of optimizing model parameters directly with respect to evaluation metrics”. Another impressive improvement comes from improving translation with monolingual data [7]. Modern neural nets translate text ∼1000 times faster than humans do [54]. Study of foreign languages is now of less practical use because NMT improves faster than most humans can.