2013 in Review: Strategic and Expository Research

This is the 3rd part of my personal and qualitative self-review of MIRI in 2013, in which I begin to review MIRI’s 2013 research activities. By “research activities” I mean to include outreach efforts primarily aimed at researchers, and also three types of research performed by MIRI:

Expository research aims to consolidate and clarify already-completed strategic research or Friendly AI research that hasn’t yet been explained with sufficient clarity or succinctness, e.g. “Intelligence Explosion: Evidence and Import” and “Robust Cooperation: A Case Study in Friendly AI Research.” (I consider this a form of “research” because it often requires significant research work to explain ideas clearly, cite relevant sources, etc.)
Strategic research aims to clarify how the future is likely to unfold, and what we can do now to nudge the future toward good outcomes, and involves more novel thought and modeling than expository research — though, the distinction is fuzzy. See e.g. “Intelligence Explosion Microeconomics” and “How We’re Predicting AI — or Failing to.”1
Friendly AI research aims to solve the technical sub-problems that seem most relevant to the challenge of designing a stably self-improving artificial intelligence with humane values. This often involves sharpening philosophical problems into math problems, and then developing the math problems into engineering problems. See e.g. “Tiling Agents for Self-Modifying AI” and “Robust Cooperation in the Prisoner’s Dilemma.”

I’ll review MIRI’s strategic and expository research in this post; my review of MIRI’s 2013 Friendly AI research will appear in a future post. For the rest of this post, I usually won’t try to distinguish which writings are “expository” vs. “strategic” research, since most of them are partially of both kinds.

Strategic and expository research in 2013

In 2013, our public-facing strategic and expository research consisted of 4 papers published directly by MIRI, 4 journal-targeted papers, 4 chapters in a peer-reviewed book, 9 in-depth analysis blog posts, 14 short analysis blog posts, and 16 interviews with domain experts.
I think these efforts largely accomplished the goals at which they were aimed, but in 2013 we learned a great deal about how to accomplish those goals more efficiently in the future. In particular…
Expert interviews seem to be the most efficient way to accomplish some of those goals.
Rather than conducting large strategic research projects ourselves, we should focus on writing up what is already known (“expository research”) and on describing open research questions so that others can examine them.

What we did in 2013 and why
Below I list the writings that constitute MIRI’s public-facing2 strategic and expository research in 2013.

4 papers published directly by MIRI: (1) Yudkowsky’s “Intelligence Explosion Microeconomics,”3 (2) Sotala & Yampolskiy’s “Responses to Catastrophic AGI Risk: A Survey,” (3) Grace’s “Algorithmic Progress in Six Domains,” and (4) Fallenstein & Mennen’s “Predicting AGI: What can we say when we know so little?“
4 journal-targeted papers, two of them published and two of them still being considered by target journals: (1) Shulman & Bostrom’s “Embryo Selection for Cognitive Enhancement,” (2) Armstrong et al.’s “Racing to the Precipice,” (3) Yampolskiy & Fox’s “Safety Engineering for Artificial General Intelligence,”4 and (4) Muehlhauser & Bostrom’s “Why We Need Friendly AI.”5
4 chapters in a peer-reviewed book from Springer called Singularity Hypotheses: A Scientific and Philosophical Assessment. Three chapters were written by MIRI staff members: “Intelligence Explosion: Evidence and Import,” “Intelligence Explosion and Machine Ethics,” and “Friendly Artificial Intelligence.”6 One additional chapter was co-authored by then-MIRI research associate Joshua Fox: “Artificial Intelligence and the Human Mental Model.” MIRI also contributed two short replies to other chapters, one reply by Yudkowsky and another by Michael Anissimov.7
9 in-depth “analysis” blog posts: (1) Sotala’s “A brief history of ethically concerned scientists,” (2) Kaas’ “Bayesian Adjustment Does Not Defeat Existential Risk Charity,” Yudkowsky’s  (3) “The Robots, AI, and Unemployment Anti-FAQ” and (4) “Pascal’s Muggle,” and Muehlhauser’s (5) “AGI Impact Experts and Friendly AI Experts,” (6) “When Will AI Be Created?“, (7) “Transparency in Safety-Critical Systems,” (8) “How effectively can we plan for future decades? (initial findings),” and (9) “How well will policy-makers handle AGI? (initial findings).”
14 short analysis blog posts: Yudkowsky’s (1) “Five theses, two lemmas, and a couple of strategic implications,” (2) “After critical event W happens, they still won’t believe you,” (3) “Do Earths with slower economic growth have a better chance at FAI?,” and (4) “Being Half-Rational About Pascal’s Wager is Even Worse,” and Muehlhauser’s (5) “Friendly AI Research as Effective Altruism,” (6) “What is intelligence?“, (7) “What is AGI?“, (8) “AI Risk and the Security Mindset,” (9) “Mathematical Proofs Improve But Don’t Guarantee Security, Safety, and Friendliness,” (10) “Richard Posner on AI Dangers,” (11) “Russell and Norvig on Friendly AI,” (12) “From Philosophy to Math to Engineering,” (13) “Intelligence Amplification and Friendly AI,” and (14) “Model Combination and Adjustment.”
16 interviews with domain experts:  (1) James Miller on unusual incentives facing AGI companies, (2) Roman Yampolskiy on AI safety engineering, (3) Nick Beckstead on the importance of the far future, (4) Benja Fallenstein on the Löbian obstacle to self-modifying systems, (5) Holden Karnofsky on transparent research analyses, (6) Stephen Hsu on Cognitive Genomics, (7) Laurent Orseau on Artificial General Intelligence, (8) Paul Rosenbloom on Cognitive Architectures, (9) Ben Goertzel on AGI as a Field, (10) Hadi Esmaeilzadeh on Dark Silicon, (11) Bas Steunebrink on Self-Reflective Programming, (12) Markus Schmidt on Risks from Novel Biotechnologies, (13) Robin Hanson on Serious Futurism, (14) Greg Morrisett on Secure and Reliable Systems, (15) Scott Aaronson on Philosophical Progress, and (16) Josef Urban on Machine Learning and Automated Reasoning.8
One recorded and transcribed conversation about effective altruism in general, with other members of the effective altruism movement: Effective Altruism and Flow-Through Effects.

MIRI staff members have varying opinions about the value and purpose of strategic and expository research. Speaking for myself, I supported or conducted the above research activities in order to:9

Test our assumptions and try to understand the views of people who (might) disagree with us. Examples: “How effectively can we plan for future decades?”, “How well will policy-makers handle AGI?”, and the Greg Morrisett interview.
Learn new things that can inform strategic action concerning existential risk and Friendly AI. Examples: “Algorithmic Progress in Six Domains,” the Hadi Esmaeilzadeh interview, and the Josef Urban interview.
Make it easier for other researchers to contribute, by performing small bits of initial work on questions of strategic significance, or by explaining how an open question in superintelligence strategy could be studied in more depth. Examples: “Intelligence Explosion Microeconomics,” “Algorithmic Progress in Six Domains,” and “How effectively can we plan for future decades?”
Build relationships with researchers who might one day contribute to strategic, expository, or Friendly AI research. Examples: many of the interviews.
Explain small “pieces of the puzzle” that contribute to MIRI-typical views about existential risk and Friendly AI. Examples: “When Will AI Be Created?”, “Mathematical Proofs Improve But Don’t Guarantee…,” and the Nick Beckstead interview.

How well did these efforts achieve their goals?
We have not yet implemented quantitative methods for measuring how well our strategic and expository research efforts are meeting the goals at which they are aimed.10 For now, I can only share my subjective, qualitative impressions, based on my own reasoning and a few conversations I had with some people who follow our research closely, after showing them a near-complete draft of the previous section.
Re: goal (1). It’s difficult to locate cheap, strong tests of our assumptions. So, the research aimed at this goal conducted in 2013 either weakly confirmed some of our assumptions (e.g. see the Greg Morrisett interview11 ) or could make only small steps toward providing good tests of our assumptions (e.g. see “How effectively can we plan for future decades?” and “How well will policy-makers handle AGI?”).
Re: goal (2). Similarly, it’s difficult to locate inexpensive evidence that robustly pins down the value of an important strategic variable (e.g. AI timelines, AI takeoff speed, or the strength of the “convergent instrumental values” attractor in mind design space). Hence, research aimed at learning new things typically only provides small updates (for us, anyway), e.g. about the prospects for Moore’s law (the Hadi Esmaeilzadeh interview) and about the current state of automated mathematical reasoning (the Josef Urban interview).
My own reaction to the difficulty of obtaining additional high-likelihood-ratio evidence about long-term AI futures goes something like this:
Well, the good news is that humanity seems to have seized most of the low-hanging fruit about future machine superintelligence, which wasn’t the case 15 years ago. The bad news is that the low-hanging fruit alone doesn’t make it clear how we go about winning. But since the stakes are really high, we just have to accept that long-term forecasting is hard, and then try harder. We need to get more researchers involved so more research can be produced, and we must be prepared to accept that it might take 10 PhD theses worth of work before we get a 2:1 Bayesian update about a strategically relevant variable. Also, it’s probably good to “marinate” one’s brain in relevant fields even if one isn’t sure which specific updates one will be able to make as a result, because filling one’s brain with facts about relevant fields will likely improve one’s intuitions in general about those fields and adjacent fields.12
Re: goal (3). I don’t have a good sense of how useful MIRI’s 2013 strategic and expository research has been for other researchers, but such effects typically require several years to materialize.13 I’m optimistic about this work enabling further research by others simply because that’s how things typically work in other fields of research, and I don’t see much reason to think that superintelligence strategy will be any different.
Re: goal (4). Yes, many of the interviews built new relationships with helpful domain experts.
Re: goal (5). Again, I don’t have good measures of the effects here, but I do receive frequent comments from community members that “such-and-such post was really clarifying.” Some of the analyses are also linked regularly by other groups. For example, both GiveWell and 80,000 Hours have linked to our model combination post when explaining their own research strategies.

Looking ahead to 2014
As discussed above and in my operations review, we still need to find better ways to measure the impact of our research. A plausible first-try measurement technique would be to survey a subset of the people we hope to impact in various ways, and ask how our research has impacted them.
Even before we can learn from improved impact measurement, however, I think I can say a few things about what I’ve learned about doing strategic and expository research, and what we plan to do differently in 2014.
First, interviews with domain experts are a highly efficient way to achieve some of the goals I have for expository and strategic research. Each interview required only a few hours of staff time, whereas a typical “short” analysis post cost between 5 and 25 person-hours, and a typical “in-depth” analysis post cost between 10 and 60 person-hours.
In 2013 we published 16 domain expert interviews between July 1st and December 30th, an average of 2.66 interviews per month. In 2014 I intend to publish 4 or more interviews per month on average.
Second, expository research tends to be more valuable per unit effort than new strategic research. MIRI (in conjunction with our collaborators at FHI) has an uncommonly large backlog of strategic research that has been “completed” but not explained clearly anywhere. Obviously, it takes less effort to explain already-completed strategic research than it takes to conduct original strategic research and then also explain it.
Third, we can prioritize expository (and sometimes strategic) research projects by dialoguing with intelligent critics who are representative of populations we want to influence (e.g. AI researchers, mega-philanthropists) and then preparing the writings most relevant to their concerns. We can then dialogue with them again after they’ve read the new exposition, and see whether that particular objection remains, and if so why, and if not then what other objections remain — which can in turn inform our prioritization of future writings, and also potentially reveal flaws in our models.
Fourth, students want to know which research projects they could do that would help clarify superintelligence strategy. Unfortunately, experienced professors are not yet knocking down our door to ask us which papers they could research and write to clarify superintelligence strategy, but many graduate students are. Also, I’ve had a few conversations with graduate student advisors who said they have to put lots of time into helping their students find good projects, and that it would be helpful if somebody else prepared research project proposals suitable for their students and their department.
Furthermore, there is some historical precedent for this strategy working, even within the young, narrow domain of superintelligence strategy. The clearest example is that of Nick Beckstead, who wrote a useful philosophy dissertation on the importance of shaping the far future, in part due to conversations with FHI. João Lourenço is currently writing a philosophy dissertation about the prospects for moral enhancement, in part due to conversations with FHI and MIRI. Jeremy Miller is in the early planning stages of a thesis project about universal measures of intelligence, in part due to conversations with MIRI. I think there are other examples, but I haven’t been able to confirm them yet.
So, in 2014 we plan to publish short descriptions of research projects which could inform superintelligence strategy. This will be much easier to do once Nick Bostrom’s Superintelligence book is published, so we’ll probably wait until that happens this summer.
Fifth, Nick Bostrom’s forthcoming scholarly monograph on machine superintelligence provides a unique opportunity to engage more researchers in superintelligence strategy. As such, some of our “outreach to potential strategic researchers” work in 2014 will consist in helping to promote Bostrom’s book. We also plan to release a reading guide for the book, to increase the frequency with which people finish, and benefit from, the book.
Note that what I call “MIRI’s strategic research” or “superintelligence strategy research” is a superintelligence-focused subset of what GiveWell would call “strategic cause selection research” and CEA would call this “cause prioritization research.”As usual, we also did significant strategic research in 2013 that is not public-facing (at least not yet), for example 100+ hours of feedback on various drafts of Nick Bostrom’s forthcoming book Superintelligence: Paths, Dangers, Strategies, 15+ hours of feedback on early drafts of Robin Hanson’s forthcoming book about whole brain emulation, and much work on forthcoming MIRI publications.Yudkowsky labeled this as “open problem in Friendly AI #1”, but I categorize it as strategic research rather than Friendly AI research.At the time of publication, Joshua Fox was a MIRI research associate.“Why We Need Friendly AI” was published in an early 2014 issue of the journal Think, but it was released online in 2013.The “Friendly Artificial Intelligence” chapter is merely an abridged version of Yudkowsky’s earlier “Artificial Intelligence as a Positive and Negative Factor in Global Risk.”These chapters were written during 2011 and 2012, but not published in the book until 2013.There were also two very short interviews with Eliezer Yudkowsky: “Yudkowsky on Logical Uncertainty” and “Yudkowsky on ‘What can we do now?’“I have an additional goal for some of our outreach and research activities, which is to address difficult problems in epistemology, because they are more relevant to MIRI’s research than to (e.g.) business or the practice of “normal science” (in the Kuhnian sense). “Pascal’s Muggle” is one example. Also, some of our expository and strategic research doubles as general outreach, e.g. the popular interview with Scott Aaronson.Well, we can share some basic web traffic data. According to Google Analytics, the pages (of 2013’s strategic or expository research) with the most “unique pageviews” since they were created are: “When will AI be created?” (~15.5k), the Scott Aaronson interview (~13.5k), the Hadi Esmaeilzadeh interview (~13.5k), “The Robots, AI, and Unemployment Anti-FAQ” (~12k), “What is intelligence?” (~5k), “Pascal’s Muggle” (~5k), “A brief history of ethically concerned scientists” (~4.5k), “Intelligence explosion microeconomics” (~3.5k), and “From philosophy to math to engineering” (~3.5k). Naturally, this list is biased in favor of articles published earlier. Also, Google Analytics doesn’t track PDF downloads, so we don’t have numbers for those.E.g. see his statements “Yes, I completely agree with [the ‘Mathematical Proofs Improve…‘ post]” and “I think re-architecting and re-coding things will almost always lead to a win in terms of security, when compared to bolt-on approaches.”This last bit is part of my motivation for listening to so many nonfiction audiobooks since September 2013.“Intelligence Explosion Microeconomics” enabled “Algorithmic Progress in Six Domains,” but it was still the case that MIRI had to commission “Algorithmic Progress in Domains.”The post 2013 in Review: Strategic and Expository Research appeared first on Machine Intelligence Research Institute.


MIRI’s Experience with Google Adwords

In late 2011, MIRI opened a Google Grants account, which provides $10k/mo in free Google Adwords for nonprofits. Kevin Fischer and I tweaked our Adwords account over several months until we successfully spent the full $10,000/mo 3 months in a row.
This qualified us for Grants Pro, and we are still grandfathered into that (now unavailable) $40k/mo level of free Adwords. The limit is actually $1350/day which is challenging to spend wisely, even with the recently increased max bidding level of $2/click. But with more tweaking we are now able to spend nearly all of it each month. Kevin and I probably spent 100 hours between us over the past few years optimizing this, but much of it was done while we were both volunteering for MIRI. Ongoing tweaking requires only an hour or less of my time per month.
The traffic is large (2/3 of our total) but marginal in quality. In the past 6 months, we drove ~250,000 visitors to MIRI’s site via Google Adwords.

5000 people read at least one of our research papers
500 signed up for the newsletter (our true goal since it gives them a chance to hear from us again)
150 went to the volunteer site (and almost all didn’t sign up once they got there)
100 applied to attend research workshops (no qualified candidates yet)

Our impression is that MIRI has an especially hard time making good use of Google Adwords, because there is such a gulf of inferential distance between what we do and what people already know. Many things we could show someone who had never heard about us to try to have a strong impact would plausibly be more misleading than helpful. We expect most charities could make higher-value use of Google Adwords than we can for this reason, including e.g. effective altruism meta-charities or animal welfare groups.
Paying $16 for each newsletter subscriber is pretty bad, and it’s not remotely how we would spend $1350/day if it was unrestricted money. But, Adwords creates some value on the margin and we’re glad Google includes us in the program and that we’re able to reach new people about our work by using it. It’s 1000 new people getting our newsletter every year, and more eyeballs on our content. Some of them might pass it to someone else who would be good for a workshop, or something.
Also a word of warning to other nonprofits: we tried many times to get interns or volunteers to improve our Adwords account, but nobody was good at it. Lots of remote supporters (some of whom swore they were amazing at AdWords) were given access and didn’t make a single change to a single campaign, much less create new experiments and find improvements. We also briefly tried paying a contractor to make new ads and they at least tried things, but their ads didn’t generate value so we had to let them go.
If charities work with volunteers or supporters to improve their Adwords accounts, I’d recommend requiring that volunteers produce proof that they are currently managing at least one other large Adwords account successfully (or only ask volunteers for cheap things like ideas and don’t expect any help from them doing the much more costly and difficult work of actually implementing their ideas).
The post MIRI’s Experience with Google Adwords appeared first on Machine Intelligence Research Institute.

MIRI’s Experience with Google Adwords


Careers at MIRI

We’ve published a new Careers page, which advertises current job openings at MIRI.
As always, we’re seeking math researchers to make progress on Friendly AI theory. If you’re interested, the next step is not to apply for the position directly, but to apply to attend a future MIRI research workshop.
We are also accepting applications for a grants manager, a science writer, and an executive assistant.
Visit our Careers page to apply.

The post Careers at MIRI appeared first on Machine Intelligence Research Institute.

Careers at MIRI


Ronald de Wolf on Quantum Computing

Ronald de Wolf is a senior researcher at CWI and a part-time full professor at the University of Amsterdam. He obtained his PhD there in 2001 with a thesis about quantum computing and communication complexity, advised by Harry Buhrman and Paul Vitanyi. Subsequently he was a postdoc at UC Berkeley. His scientific interests include quantum computing, complexity theory, and learning theory.
He also holds a Master’s degree in philosophy (where his thesis was about Kolmogorov complexity and Occam’s razor), and enjoys classical music and literature.

Luke Muehlhauser: Before we get to quantum computing, let me ask you about philosophy. Among other topics, your MSc thesis discusses the relevance of computational learning theory to philosophical debates about Occam’s razor, which is the principle advocating that “among the theories, hypotheses, or explanations that are consistent with the facts, we are to prefer simpler over more complex ones.”
Though many philosophers and scientists adhere to the principle of Occam’s razor, it is often left ambiguous exactly what is meant by “simpler,” and also why this principle is justified in the first place. But in your thesis you write that “in certain formal settings we can, more or less, prove that certain versions of Occam’s Razor work.”
Philosophers are usually skeptical when I argue for K-complexity versions of Occam’s razor, as you do. For example, USC’s Kenny Easwaran once wrote, “I’ve never actually seen how [a K-complexity based simplicity measure] is supposed to solve anything, given that it always depends on a choice of universal machine.”
How would you reply, given your optimism about justifying Occam’s razor “in certain formal settings”?

Ronald de Wolf: I would treat Occam’s razor more as a rule of thumb than as a formal rule or theorem. Clearly it’s vague, and clearly there are cases where it doesn’t work. Still, many scientists have been guided by it to good effect, often equating simplicity with beauty (for example Einstein and Dirac). Psychologically, invoking Occam will only be effective if there is some shared notion of simplicity; maybe not to quantify simplicity, but at least to be able to rank theories according to their simplicity.
You could try to use Kolmogorov complexity as your “objective” measure of simplicity, and in some simplified cases this makes perfect sense. In my MSc thesis I surveyed a few known cases where it provably does. However, such cases do not provide convincing proof of Occam’s razor “in the real world”. They are more like thought experiments, where you strip away everything that’s superfluous in order to bring out a certain point more clearly.
In practice there are at least three issues with using Kolmogorov complexity to measure simplicity. First, it requires you to write down your theory (or whatever it is whose simplicity you’re quantifying) over some fixed alphabet, say as a string of bits. It’s often kind of subjective which background assumptions to count as actually part of your theory. Second, as Easwaran rightly says, KC depends on the choice of universal Turing machine w.r.t. which it is defined. However, I don’t think this is such a big issue. If you choose some reasonably efficient universal Turing machine and consider the KC of reasonably long strings, the constant difference incurred by the choice of universal Turing machine will be relatively small. Thirdly and possibly most importantly, KC is not computable, not even approximable by any computational process (even a very slow one) with any approximation-guarantees. This rules out using KC itself in practical settings.
However, the core idea that compression somehow corresponds to detection of patterns in your data is a perfectly valid one, and you can use it in practice if you’re willing to base “compression” on imperfect but practical programs like gzip. This loses the theoretical optimality guaranteed by KC (which you can view as the “ultimate compression”) but it gives you a tool for data mining and clustering that’s often quite good in practice. See for example here. Such practical approaches are like heuristics that try to approach, in some weak sense, the ideal but unreachable limit-case of KC.

Luke: Do you think one can use Occam-like principles to choose between, for example, the various explanations of quantum mechanics, since they appear to make essentially the same predictions about what we should observe?

Ronald: In principle you could, but to my limited understanding (I’m not following this debate closely), the main interpretations of QM all suffer from having some seemingly superfluous aspects. The standard interpretation that a measurement “collapses the wave function” to a probabilistically-chosen measurement outcome treats “observers” as a special category of quantum objects, or “observation/measurement” as a special category of quantum process. Before you know it, people will bring consciousness into the picture and mysticism beckons. It seems to me that treating the “observer” as a special category violates Occam’s razor. Alternatively you can take the position that measurement is nothing special but just another interaction between quantum systems (observer and observed system). This is sometimes known as the “church of the larger Hilbert space”. It’s mathematically pleasing because now there’s only this smooth, coherent, and even deterministic evolution of the whole universe. However, now you will have may different “branches” of the superposition that is the world’s state vector, which very quickly leads to the multiverse view of many worlds. A metaphysics that postulates infinitely many worlds existing in superposition doesn’t strike me as very Occam-compliant either.
Then there is the instrumentalist “shut up and calculate” school. This is minimalistic in an Occam-pleasing sense, but seems to substantially impoverish the scientific endeavour, whose aim should not just be to predict but also to explain and give some picture of the world. All interpretations of QM are problematic in their own way, and choosing between them based on Occam’s razor assumes some shared idea of what simplicity is as well as a shared view of the goals of science, which we seem to lack here.

Luke: Most of your work these days is in quantum computing and communication. Quantum computing is an interesting field, since its researchers design algorithms, error correction techniques, etc. for machines that cannot yet be built. In this sense, I tend to think of quantum computing as an “exploratory engineering” discipline, akin to pre-Sputnik astronautics, pre-ENIAC computer science, and Eric Drexler’s Nanosystems. Do you think that’s a fair characterization? Do you or your colleagues in quantum computing get much criticism that such work is “too speculative”? (For the record, that’s not my view.)

Ronald: The two main questions in quantum computing are (1) can we build a large-scale quantum computer and (2) what could it do if we had one. I think your term “exploratory engineering” fits the work on the first question; small quantum computers on a handful of qubits were already built a decade ago, so it’s not pure theory anymore. I myself am a theoretical computer scientist focusing on the second question. While I think this is more mathematics than engineering, you can certainly compare it to computer science in the 1930s: at that point the theoretical model of a (classical) computer had already been introduced by Alan Turing, but no large-scale computers had been built yet. You could already design algorithms for Turing machines on paper, and you could even prove that such computers could not solve certain problems (as Turing famously did for the halting problem). We are doing such work on quantum computing now: designing quantum algorithms and communication protocols that are much faster than classical problems for some computational problems, and on the other hand proving that quantum computers do not give you a speed-up for many other problems. Much of the relevance of this is of course contingent upon the eventual construction of a large QC. Interestingly, however, some of the work we are doing has spin-offs for the analysis of classical computing, and that is relevant today irrespective of progress on building a QC.
Regarding the possible charge of being “too speculative”: in the mid-1990s, right after Peter Shor published his groundbreaking quantum algorithm for factoring large numbers into their prime factors (which breaks a lot of cryptography), there was a lot of skepticism, particularly among physicists who thought that this would never fly. They expected that any attempt at implementing quantum bits and operations would have so much noise and errors that it would quickly decohere to a classical computer. Of course they had good reasons to be skeptical — manipulating something as small as an electron is extremely hard, much harder than manipulating a vacuum tube was in the 1940s and 1950s. The worries about noise and imperfections were partially answered soon after by the development (partially by Shor himself) of quantum error-correction and fault-tolerant computing, which roughly says that if the noise is not too large and not too malicious, your quantum computer can correct for it. The only way these worries can be fully overcome is to actually build a large-scale QC. My impression is that experimental physicists are making slow but sure progress on this, and are becoming more optimistic over time that this will actually be realized within one or two decades. So, sure this is a speculative endeavour (most long-term research is), but not unreasonably so.

Luke: What heuristics do you and your colleagues in quantum computing use to decide what to work on, given QC’s long-term and somewhat speculative nature? Presumably you need to make uncertain predictions about which types of quantum computers are most likely to be built, what the solutions to known obstacles might look like, etc.? (I ask because MIRI aims to conduct long-term research that is more speculative than quantum computing.)

Ronald: Most of the time we study how well quantum computers can solve classical computational problems, problems with classical inputs (such as a large number N) and classical outputs (such as the prime factors of N). Computer science has over decades been defining and studying the complexity of lots of interesting and useful computational problems and models, and often we start from there: we take an existing computational problem and try to find quantum tricks to improve over the best classical solutions. In some cases we succeed, designing quantum ways to outperform classical computers, and in some cases we can prove that a QC can’t do better than a classical computer. Of course it’s hard to predict what quantum tricks (if any) might help for a specific problem, but we have some general tools at our disposal. For example, quantum computers are good at detecting periodic patterns (that’s the core of Shor’s algorithm); they can search faster (Grover’s algorithm); you can hide information by encoding it in an unknown basis (quantum cryptography); you can pack a doubly-exponential number of quantum states in an n-qubit space (quantum fingerprinting), etc. A lot of work is based on skillfully combining and applying such known quantum tools, and once in a while people find new tricks to add to our toolbox. Of course, there is also work of a more specific quantum nature, which is not just throwing quantum tricks at classical problems. For example, a lot of work has been done recently on testing whether given quantum states are properly entangled (and hence can be used, for instance, in quantum cryptography).
We typically abstract away from the details of the specific physical system that will implement the quantum computer. Instead we just focus on the mathematical model, with quantum bits and a well-defined set of elementary operations (“gates”) that we can perform on them. It doesn’t really matter whether the qubits will be implemented as electron spins, or as photon polarizations, or as energy levels of an atom — from the perspective of the model, it only matters that a qubit has well-defined 0 and 1 states and that we can form superpositions thereof. Similarly, for classical computers it doesn’t really matter whether you program in C or Java or assembler; all such programming languages can efficiently simulate each other. And you don’t care about the precise voltages used to implement bits physically, as long as each bit has stable and clearly distinguished 0 and 1 values.
Abstracting away from such implementation details is justified when we have a large-scale quantum computer, because different varieties of quantum computers will be able to simulate each other with only moderate overhead in terms of additional number of qubits and operations needed. For example, for the purposes of designing quantum algorithms it’s convenient to assume that you can interact any pair of qubits, even when they are far apart; in the reality of physical experiments it’s much simpler to allow only nearest-neighbor interactions between qubits. We can design algorithms for the first model and then implement them in the nearest-neighbor model by inserting a few swap-operations to move interacting qubits close together. However, this “moderate overhead” is actually quite significant as long as we do not yet have a large-scale quantum computer. It’s quite likely that on the slow road towards a large QC we will first have QCs with a few dozen or a few hundred qubits (the current state of the art is a few qubits). In this case we can’t be too wasteful and probably should design algorithms that are optimized for specific physical implementations. It is actually a very interesting question to find problems where a 50- or 100-qubit QC can already outperform classical computers in some noticeable way. Such problems would be the benchmark on which intermediate-size QCs could be tested.
The point is that once you have a large number of qubits available, the differences between different physical implementations/architectures don’t matter too much, because they are all equivalent up to small overheads (needed to simulate one variant using another). But when we have only intermediate-size QCs available (of, say, a few dozen or a few hundred qubits), then these overheads do make a big difference, and we need to carefully optimize our quantum algorithm for performing on the specific physical implementation that’s actually available. In this respect quantum computing seems quite different from most other future technologies: somehow we’re better able to predict the power of this technology for the long term (when we’ll hopefully have a large-scale QC available and can essentially ignore implementation details) than for the short and medium term (while we only have small-scale QCs with quirky limitations).

Luke: My next question leaps from quantum computing to technological forecasting. What is your subjective probability that we’ll have a 500-qubit quantum computer, which is uncontroversially a quantum computer, within the next 20 years? And, how do you reason about a question like that?

Ronald: Quite high, let’s say probability greater than 2/3. That’s the typical computer science threshold for a “bounded-error” algorithm. From a theoretical perspective, I don’t think we know of any fundamental obstacles to building a large-scale QC, and the threshold theorem from fault-tolerant QC assures us we can deal with moderate amounts of noise and errors. Clearly building a QC is an exceedingly hard engineering problem, but my impression is that experimentalists are making slow but sure progress. There are basically three possible scenarios here:

Someone constructs a large QC
We discover a fundamental problem with quantum mechanics (which would be extremely interesting new physics!)
Experimentalists muddle through without too much progress until either they or the funding agencies lose faith and give up.

The first scenario seems the most plausible to me. I should qualify this by saying that I’m not a certified physicist, let alone a certified experimental physicist, so this opinion is partly based on hearsay — but I do have some confidence in the progress that’s happening in places like MIT, NIST, Yale, Delft,… The recent paper you refer to casts doubt upon the controversial D-Wave quantum computer, which has gotten a lot of press in the last few years. For commercial reasons they prioritize quantity (=number of available qubits) over quality (=the coherence and “true quantum nature” of those qubits), and their machines seem too noisy to have useful quantum computing power.

Luke: Does that mean we probably need to purge Earth of Shor-breakable crypto-security, and transition to post-quantum cryptography, within ~20 years?

Ronald: I think that would be a wise precaution, at least for important or sensitive data. There are at least two ways to handle this. We could either stick with public-key cryptography but replace Shor-breakable problems like factoring and discrete logs by problems that seem to be hard to crack even for QC; lattice problems are an oft-mentioned candidate. Or we could use quantum cryptography. Neither is as efficient as RSA, but at least they’re more secure. It makes sense to start this transition already now, even though there’s no QC yet: the security services (and, who knows, maybe the mafia too) are probably hoovering up RSA-encrypted communications that they store for the time being, waiting for the QC that will allow them to decrypt these messages later. So even today’s communication is not safe from a future QC.

Luke: Thanks, Ronald!
The post Ronald de Wolf on Quantum Computing appeared first on Machine Intelligence Research Institute.

Ronald de Wolf on Quantum Computing


Robust Cooperation: A Case Study in Friendly AI Research

The paper “Robust Cooperation in the Prisoner’s Dilemma: Program Equilibrium via Provability Logic” is among the clearer examples of theoretical progress produced by explicitly FAI-related research goals. What can we learn from this case study in Friendly AI research? How were the results obtained? How did the ideas build on each other? Who contributed which pieces? Which kinds of synergies mattered?
To answer these questions, I spoke to many of the people who contributed to the “robust cooperation” result.
I’ll begin the story in December 2011, when Vladimir Slepnev (a Google engineer in Zurich) posted A model of UDT with a halting oracle, representing joint work with Vladimir Nesov (a computer science grad student in Moscow).1 This post, arguably for the first time,2 presented a formal model of Wei Dai’s updateless decision theory (UDT) and showed that UDT agents would “win” when presented with Newcomb’s Problem — if the universe program and its agent sub-program had access to a halting oracle, anyway. Nisan Stiennon (a math grad student at Stanford) applied Slepnev’s formalization to the problem of proving cooperation using Peano Arithmetic in Formulas of arithmetic that behave like decision agents (Feb. 2012).3
The success of these two posts in formalizing UDT inspired Patrick LaVictoire (a math postdoc in Madison) to attempt a “semi-formal analysis” of timeless decision theory (TDT), an earlier decision theory invented by Eliezer Yudkowsky (MIRI’s founder), which itself had been a significant inspiration for UDT. After three setup posts, LaVictoire thought he had successfully formalized something kind of like TDT in April 2012.
LaVictoire didn’t get much reaction from other TDT/UDT researchers, so when he visited the SF Bay Area for a CFAR workshop in July 2012, he tracked down Yudkowsky, Stiennon, Paul Christiano (a computer science grad student at Berkeley), and several others to talk to them about his attempt to formalize TDT. Their reaction was positive enough that LaVictoire was encouraged to keep working on the approach.
LaVictoire also discussed his work with Slepnev when Slepnev visited the Bay Area in August 2012, and Slepnev pointed out that LaVictoire’s attempted formalization of TDT (now called “Masquerade”) was fatally flawed for Löbian reasons. But in September 2012, LaVictoire was able to patch the problem by having Masquerade escalate between different formal systems. At this point, LaVictoire began writing an early draft of the “Robust Cooperation” paper.
Slepnev insisted on the importance of optimality results, so later that month LaVictoire came up with a candidate optimality notion, and then in October noticed that Masquerade itself failed to be optimal by that definition. This was roughly the state of things when MIRI’s April 2013 workshop began.
Early in the workshop, LaVictoire gave a Masquerade tutorial to the other participants. Tweaking Masquerade eventually led to the concept of modal agents, and LaVictoire and Mihály Barasz (a Google engineer in Zurich) began looking for ways to mechanically validate the actions of such agents against one another. Eventually, Barasz and Marcello Herreshoff (a Google engineer in the Bay Area) developed a model checker for modal agent interactions, so that agents’ choices against other agents could be proved mechanically.
Near the end of the April workshop, Christiano developed PrudentBot, which is in some sense the “star” of the current paper. Additional contributions were made by Yudkowsky, Benja Fallenstein (a grad student at Bristol University), and others during the workshop. LaVictoire updated the draft paper with the April workshop’s results, and posted it to Less Wrong in June 2013.
Later, at MIRI’s September 2013 workshop, Kenny Easwaran (a philosopher at USC) found that it was harder than LaVictoire had expected to prove that any unexploitable agent must eventually fail to optimize against some kind of WaitFairBot. Herreshoff worked on patching this, but the proof was ballooning that section of the paper beyond recognition for a minor result, so LaVictoire decided to remove it from the paper.
In December 2013, Fallenstein found that the paper didn’t adequately show that the actions of two modal agents depend solely on their modal description, and he introduced a set of patches for this. LaVictoire modified the paper once more, and then, with the consent of his co-authors, uploaded the revised paper to arxiv in January 2014.
What, then, is the meaning and significance of the results in the “Robust Cooperation” paper? LaVictoire’s view, at least, is this:
The significance of modal combat is that it’s a toy universe in which we can study concepts of advanced decision theory (and which we might modify slightly in order to study other concepts, like blackmail), and within which the intuitively appealing idea of superrationality in fact works out. It’s at least a philosophical hint that good communication can enable cooperation without the usual costs of enforcement and punishment, and that there are incentives toward simplicity and verifiability among rational agents.
In fact, it’s an even more basic analogue of an Iterated Prisoner’s Dilemma tournament. Just as Axelrod’s IPD [iterated PD] tournament illustrated the usefulness of “tough but fair” and gave rise to the idea of evolutionary incentives for reciprocal altruism, I think that modal combat is a useful sandbox for illustrating the logic of “superrationality.” Furthermore, modal combat includes many of the features of the IPD (with levels of deduction being somewhat analogous to an agent’s historical interactions with another agent), and it has extremely simple grammar for the level of sophistication of these algorithms.
The development of updateless decision theory itself is another story, which won’t be recounted here in detail. Two brief sources on this story are the ‘Prior Work’ section of Vladimir Nesov’s Controlling Constant Programs, and also this comment. Nesov’s very brief summary of the development of UDT goes like this: “(1) Eliezer Yudkowsky’s early informal remarks about TDT and Anna Salamon’s post brought up the point that some situations should be modeled by unusual dependencies, motivating the question of how to select an appropriate model (infer dependencies). (2) Wei Dai’s UDT post sketched one way of doing that, but at the time I didn’t understand that post as answering this question, and eventually figured it out for programs-control-programs case in May 2010. After discussion on the decision theory mailing list, Vladimir Slepnev applied the technique to the prisoner’s dilemma (PD). (3) The more general technique was then written up by Slepnev and me, with Slepnev’s posts having more technical substance, and mine more speculative, attempting to find better ways of framing the theory: What a reduction of ‘could’ could look like, Controlling constant programs, and Notion of preference in ambient control. (4) There were still a number of technical issues around ‘spurious moral arguments’. See this comment by Benja Fallenstein and An example of self-fulfilling spurious proofs in UDT. (5) One solution was adding a ‘chicken rule’ to the decision algorithm, which I figured out for the programs-control-programs case in April 2011 and discussed a bit on the decision theory list, but which turned out to be much more theoretically robust in a setting with a halting oracle, which came up in another discussion on the decision theory list in December 2011, which Slepnev wrote up in A model of UDT with a halting oracle. My later write up was Predictability of Decisions and the Diagonal Method. (6) Armed with the diagonal trick (chicken rule), Stiennon wrote up cooperation in PD for the oracle case, which was more theoretically tractable than Slepnev’s earlier no-oracle solution for PD. (7) At this point, we had both a formalization of UDT that didn’t suffer from the spurious proofs problem, and an illustration of how it can be applied to a non-trivial problem like PD.”Some researchers might instead say that Slepnev’s August 2010 post “What a reduction of ‘could’ could look like” presented the “first formal model” of UDT.Stiennon’s post also improved the formalization by using a two-step “chicken rule” rather than a one-step chicken rule.The post Robust Cooperation: A Case Study in Friendly AI Research appeared first on Machine Intelligence Research Institute.


Two MIRI talks from AGI-11

Thanks in part to the volunteers at MIRI Volunteers, we can now release the videos, slides, and transcripts for two talks delivered at AGI-11. Both talks represent joint work by Anna Salamon and Carl Shulman, who were MIRI staff at the time (back when MIRI was known as the “Singularity Institute”):
Salamon & Shulman (2011). Whole brain emulation as a platform for creating safe AGI. [Video] [Slides] [Transcript]
Shulman & Salamon (2011). Risk-averse preferences as an AGI safety technique. [Video] [Slides] [Transcript]
The post Two MIRI talks from AGI-11 appeared first on Machine Intelligence Research Institute.

Two MIRI talks from AGI-11


Mike Frank on reversible computing

Michael P. Frank received his Bachelor of Science degree in Symbolic Systems from Stanford University in 1991, and his Master of Science and Doctor of Philosophy degrees in Electrical Engineering and Computer Science from the Massachusetts Institute of Technology in 1994 and 1999 respectively. While at Stanford, he helped his team win the world championship in the 1990-91 International Collegiate Programming Competition sponsored by the Association for Computing Machinery. Over the course of his student years, he held research internships at IBM’s T.J. Watson Research Center, NASA’s Ames Research Center, NEC Research Institute, Stanford Research Institute, and the Center for Study of Language and Information at Stanford. He also spent the summer after his Freshman year as a software engineering intern at Microsoft. During 1998-1999, Mike stopped out of school for a year to work at a friend’s web startup (Stockmaster.com).
After graduation, he worked as a tenure-track Assistant Professor in the Computer and Information Science and Engineering department at the University of Florida from 1999-2004, and at the Electrical and Computer Engineering department at the Florida A&M University – Florida State University College of Engineering from 2004-2007. After an ill-fated attempt to start a business in 2007-2008, he returned to academia in a variety of short-term research and teaching positions in the Florida A&M Department of Physics and the FAMU-FSU College of Engineering. His present title is Associate in Engineering, and he spends most of his time supervising multidisciplinary senior engineering projects. Over the years, Dr. Frank’s research interests have spanned a number of different areas, including decision-theoretic artificial intelligence, DNA computing, reversible and quantum computing, market-based computing, secure election systems, and digital cash.

Luke Muehlhauser: Some long-term computing forecasts include the possibility of nanoscale computing, but efficient computing at that scale appears to require reversible computing due to the Landauer limit. Could you please explain what reversible computing is, and why it appears to be necessary for efficient computing beyond a certain point of miniaturization?

Mike Frank: Reversible computing refers to computation using logically reversible (invertible) transformations of the digital state of the machine, transformations that are also carried out using physical mechanisms that are (almost) thermodynamically reversible – that is, mechanisms that produce no, or negligibly small amounts of, physical entropy. Thermodynamic reversibility requires logical reversibility, since, if you carry out many-to-one transformations of the logical state, that necessitates a corresponding one-to-many transformation of the rest of the detailed physical state – or in other words, entropy production.
Reversible computing becomes important at a small scale because bit energies are becoming smaller, and are approaching the point (around 1 electron-volt) where they will not be able to be made much smaller without running into severe reliability problems due to thermal noise. But when you manipulate bits in an irreversible fashion, this essentially involves dissipating the entire bit energy to waste heat on each operation. Improvements in computer energy efficiency in the past have been driven mainly by reductions in bit energies, so if bit energy stops decreasing, it’s a big problem – the energy efficiency of traditional irreversible technology will level off and stop improving. Note this observation depends only on very fundamental thermodynamic considerations of thermal noise, and so it’s not affected by the details of whatever clever alternative nanoscale devices you may come up with. Your device technology could be based on superconductors, carbon nanotubes, nanophotonics, quantum interference transistors, I don’t care. As long as it’s a reliable, irreversible device, it will have to dissipate at least on the order of 1 electron volt (about 40 kT) per bit-operation in a room-temperature envronment, full stop, end of story.
With reversible computing, the situation is markedly different. Although it remains true that bit-energies in reliable devices are forced to level off in the 10s of kTs, if the device is reversible, then we no longer need to dissipate that entire bit energy on each operation. That’s the critical difference. In principle, for bit-operations that are performed using increasingly high-quality reversible transformations, the amount of energy that’s dissipated per operation can be made arbitrarily small; smaller than kT, much smaller even than kT ln 2 or 0.69 kT, which is the Landauer limit, the energy dissipation that’s fundamentally associated with the creation of a single bit’s worth of entropy. That’s the theoretical prediction, at least. But to actually realize practical, high-quality, high-performance, cost-efficient reversible computing below the Landauer limit is, I would say, one of the most difficult, hard-core engineering challenges of our time. That’s not to say it’s impossible; indeed, there has never been any valid proof from fundamental physics that it’s impossible, and to the contrary, there are many indications, from the research that’s been done to date, that suggest that it will probably be possible to achieve eventually. But it’s certainly not an easy thing to accomplish, given the state of engineering know-how at this point in time. A future computer technology that actually achieves high-quality, efficient reversible computing will require a level of device engineering that’s so precise and sophisticated that it will make today’s top-of-the-line device technologies seem as crude in comparison, to future eyes, as the practice of chiseling stone tablets looks to us today.

Luke: You write that “there has never been any valid proof from fundamental physics that [reversible computing is] impossible, and to the contrary, there are many indications… that suggest that it will probably be possible to achieve eventually.”
What are some of these indications suggesting that reversible computing should be possible to achieve eventually?

Mike: Yes, good question. First, there have been not only simulations, but also laboratory demonstrations illustrating that indeed, just as per the theoretical predictions, adiabatic switching1 of the voltage on a capacitor (for example) can indeed dissipate as little energy as desired, as the transition is carried out more slowly. However, these kinds of examples typically invoke external resonators to drive the transitions, and so this would not really count as a complete, self-contained system, until the design of those resonators is further fleshed out; but the design of high-quality resonators is a quite difficult engineering problem by itself.
In the meantime, we do already know of other self-contained systems that do undergo at least simple state transitions with negligible dissipation – consider, for example, a small, isolated diamond crystal or other solid nanostructure that is rotating or vibrating, in vacuum, under zero-gravity conditions. In principle, even in total isolation, such a structure will emit (extremely weak) gravity waves, and will therefore eventually settle into a stationary state (with no detectable vibration, and no spin except around axes around which there is perfect axial symmetry). But, for a small, rigid object, this gravitational settling process could take many billions of cycles (at least!), which, for a nanoscale object, can translate into far less than kT energy dissipation per cycle.
A related notion is that of a time crystal; these have recently been proposed as hypothetical examples of quantum systems that could theoretically cycle through a set of distinguishable states forever, even when fully relaxed into their quantum ground states. Time crystals themselves are a rather controversial concept. However, even if a true (i.e., perfect) time crystal turns out to be impossible, a system like a rotating solid object that takes a very long time to settle into a stationary state is definitely a realistic one.
The problem with that thought experiment is that a simple periodic motion, like that of a rotating or vibrating crystal, is not interesting computationally. For computational purposes, we want machines that traverse a very complexly-shaped trajectory through their configuration space, not just a short cycle. The configuration space of any useful computer has an enormous number of dimensions, but suppose, for the moment, that it had only three dimensions. Imagine now a polished ball, or particle (representing the current state of the machine), coasting along a hollowed-out channel through a solid block of material; the channel represents the constraints imposed by the structure of the computational mechanism (i.e., by the device physics). A complex reversible computation then corresponds, in this analogy, to the ball coasting smoothly, ballistically, along a long, twisty path through the material.
Can a physical system be engineered in such a way that the ball-particle (machine state) can coast through many twists and turns (many steps of computation) without losing a significant fraction of its initial dynamical (kinetic) energy? I feel that this is the fundamental question that the science of reversible computing has yet to answer definitively, one way or the other.
A good starting point would be to demonstrate any physical system that coasts along a long, complexly-shaped, deterministic (non-bifurcating) trajectory through configuration space – never mind what function it’s computing. One can always say, it’s just computing its own next state. It’s not clear why this kind of behavior should be impossible – even an everyday amusement-park roller-coaster illustrates that even macroscopic objects can coast through somewhat complex deterministic trajectories for at least a little while. And at the nanoscale, even the uncertainty principle only limits your certainty about when you arrive at the final state, not necessarily about the shape of the trajectory that you follow.
So, how far can we push this “roller-coaster” idea of ballistic computing? Maybe pretty far. But, until we have further developed a scientific and engineering discipline showing exactly how increasingly-efficient examples of such “roller-coaster” like constructs can be implemented, and wherein such constructs are designed to traverse not just a trajectory in simple 3D space, but rather in the multidimensional configuration space of a complex, nanostructured system composed of many interacting subsystems, which then might be doing something interesting in terms of computation, we will not know for sure that highly efficient reversible computing is really possible.
I personally expect that it is – if only for the lack of any sound proof that it is not (despite many attempts to rule it out having been made) – but, to really flesh out the discipline of how to engineer such systems is far more difficult, I now believe, than many of the field’s early enthusiasts may have anticipated.
What we really need, I think, are certain key theoretical breakthroughs, such as, for starters, a complete theoretical (but also realizable, i.e., not overly idealized) model of a self-contained quantum system with many interacting parts that dynamically evolves in such a way (that is, along a complex deterministic trajectory in configuration space).
I speculate that perhaps some sort of dynamical version of the quantum watchdog effect will be required in order to keep the various interacting components continuously in sync with each other, while they are simultaneously also coasting forward along complex trajectories in configuration space which are constrained, by the shape of the subsystems’ mutual interactions, to carry out interesting computations. But, I don’t know for sure if this “dynamical quantum watchdog” approach can be made to work.
Overall, to show how to fully realize reversible computing is a very thorny theoretical problem, one that I think will require serious attention from some of the foremost minds in quantum physics to solve definitively. If I had the research support, I might go back to school, bone up on my quantum theory, and try to solve it myself. But unfortunately, basic research funding for this field has been sorely lacking, in my experience, and nowadays, I have a family to support.

Luke: Why do you think reversible computing has been unable thus far to attract significant research funding and researcher time?

Mike: That is a good question. Part of the reason, I think, is that there has been a lot of misinformation and misconceptions that have circulated around about this field for a while. For example, there is a sort of scholarly rumor or “urban legend” circulating around claiming that John von Neumann, a famous pioneer of computer architecture and quantum theory, had proved at some point that information processing with less than kT ln 2 dissipation per binary “decision” was impossible. But, there is no real substance behind this legend; all that we actually have from von Neumann on this is one brief remark he made during a lecture that was not backed up by any analysis. He never published any peer-reviewed journal article on this topic, possibly because he realized that it was a mistake and had never actually proved it. Probably he was implicitly assuming that decision-making implied the creation of entropy, since an unknown outcome (of the decision) was being replaced by a known one, which would imply entropy creation elsewhere to satisfy the 2nd law of thermodynamics. But of course, a decision-making process can be deterministic, as most computations are; if the outcome is predetermined by the input, then there need be no change in entropy in the decision-making process. And, even if you want the outcome of the decision-making process to be random, that only requires moving a bit of entropy from the environment into the computer, not generating any new entropy. My feeling is that von Neumann simply hadn’t thought all this through carefully enough when he first made that remark, and that, if he had ever been exposed to the concept of reversible computing during his lifetime, he would have immediately said, “Oh, of course.”
A second rumor or urban legend against reversible computing is the claim that it would violate the central theorems proved by Claude Shannon, pioneer of the mathematical theory of information and communication, concerning the minimum power required for communication, at a given bandwidth and reliability. However, I personally have meticulously searched every single page of Shannon’s entire collected works, and absolutely nowhere in his published work did he ever address power dissipation, not even once!  Absolutely all of his work only addresses power transmitted, but nowhere do his theorems establish that the power contained in a signal cannot later be recovered by suitable mechanisms. In the design of reversible computing mechanisms, we explicitly show how the energy contained in any given pattern of physically encoded bits can be recovered, for example, by using the time-reverse of the exact same reversible process that created that bit-string in the first place. Again, if Shannon were alive today, I’m sure he would look at the principles of reversible computing and say, “Yes, of course that works.”
So, there is a bias against reversible computing that is based on the widespread misunderstanding or misinterpretation of the work of these respected pioneers. Stacked on top of this, there have been many more recent attempts to prove reversible computing impossible, but all of the supposed “proofs” contain either fallacies in reasoning or invalid assumptions, which have been shown definitively to be incorrect – I have a list of about 15 of these erroneous arguments, and the results showing that they’re clearly wrong, in one of my talks. But, each time the flaw in one of these skeptical arguments is pointed out, disproving its core objection definitively using a clear counterexample, the skeptics just keep coming up with new and different (but still flawed) arguments against reversible computing. Some of the skeptics are perhaps motivated by the desire not to lose face and admit that they’ve been mistaken in their previous arguments. I think that probably, the only way that many of these reversible computing skeptics will ever be convinced is if an already commercially-successful reversible computer is staring them in the faces – then they will no longer be able to deny it. A mere theoretical model will never satisfy the hard-core skeptics – and perhaps that’s as it should be. After all, all of the theory in the world is useless until we are able to use it to start building practical machines. And to be fair, the challenges that still need to be faced in order to make reversible computing practical are substantial. But, the prevalent attitude does make it very difficult to get research funding. And without substantial funding – or top-level researchers who have the free time and inclination to solve the remaining foundational problems definitively – these problems will never be solved, and reversible computing will never become more than an obscure academic curiosity. But, if adequate attention is paid to solving the key remaining problems, reversible computing has the potential to become the foundation of 21st century computer engineering. Certainly, we cannot make more than a modest amount of further progress in the power-performance of most computing applications without it.

Luke: Can you say a bit about the history of reversible computing thus far? What were the key pieces of conceptual progress after the initial demonstration of the Landauer limit? How did they build on each other, or on other results in physics or computing?

Mike: I’ll summarize a few of the major historical developments; but keep in mind, any brief history is bound to be incomplete. The first major theoretical development in reversible computing after Landauer’s analysis was his protege Charlie Bennett’s 1973 paper showing conclusively that irreversible operations are not strictly required for computation, that is, any computation can be carried out by use of only reversible transformations of the logical state of the machine. At worst, you might need to irreversibly initialize, just once, whatever new storage space you might want to permanently dedicate to preserve the final output of the computation – but, all of the intermediate calculations can be carried out reversibly in temporary space which can be reused by other computations. The basic technique basically involves temporarily storing all of the intermediate data that would otherwise be thrown away, so it is somewhat space-intensive, but later, in 1989, Bennett showed that more space-efficient variations on the technique were possible. Finally, it was even shown in 1997 (by Lange et al.) that reversible computing in linear space (the same as an irreversible machine) is possible, although that method is not time-efficient. It is probably the case that general reversible computations do require some amount of overhead in either space or time complexity; indeed, Ammer and I proved rigorously that this is true in a certain limited technical context. But, the overheads of reversible algorithms can theoretically be overwhelmed by their energy-efficiency benefits, to improve overall cost-performance for large-scale computations.
In terms of practical implementations, some early conceptual work was done by Fredkin and Toffoli at MIT in a 1978 proposal suggesting the use of inductors and capacitors to shuttle energy around between devices in a circuit to implement logic in a (near) dissipationless fashion. In the mid-80s, Charles Seitz and colleagues at Caltech came up with an improved technique in which inductors could be shared between logic elements and brought off-chip, but they didn’t answer the question of whether any arbitrary logical function could be implemented using their technique. In the early 1990s, Koller and Athas developed a general adiabatic logic family, but it was only able to handle combinational (not sequential) logic. Finally, in 1993 Younis and Knight at MIT developed their general-purpose, adiabatic, sequential Split-Level Charge Recovery Logic – although even that one still contained a small bug limiting its energy efficiency, which we found and fixed in the late 90s, during my thesis work. I also invented another, simpler (and bug-free) universal adiabatic logic family called 2LAL at the University of Florida in 20002. Of course, by then, many, many other research groups were also pursuing adiabatic logic, so I won’t attempt to credit all of the many contributions to the field that have been made in the meantime. However, I will say that most of the published techniques that I’ve seen for implementing adiabatic logic in CMOS (that is, using standard transistors) contain conceptual flaws of one kind or another which unnecessarily limit their energy efficiency. I have not seen any other CMOS-based approach yet that is as efficient as 2LAL. In detailed simulations at UF, we found that3 sequential 2LAL circuits could dissipate as little as 1 electron volt of energy per transistor per cycle, limited only by the leakage current in the particular device family we were simulating. This figure could be made even lower if other device families with reduced leakage were used.
People have also proposed alternative implementations of reversible computing not using transistors at all; for example, there is Craig Lent’s work at Notre Dame on quantum-dot cellular automata, and there is research that’s been done by several groups on reversible logic in superconducting circuits4. However, these alternative approaches also present their own challenges; and I’d say it’s not yet clear whether they will become viable successors to CMOS; but, at least they are trying, and are paying attention to these issues! Whereas, the vast majority of other nanocomputing proposals that have been made, namely, all of the ones that totally ignore reversible computing, are all fundamentally doomed to failure, in that they will never be able to get many orders of magnitude beyond the limits of CMOS in terms of their energy efficiency – since they all ignore the fundamental limit on the energy efficiency of irreversible devices that follows from the requirement to maintain high reliability despite thermal noise. If you want to develop a new logic technology that has any hope of being successful, in the sense of having power-performance that can potentially scale many orders of magnitude (and not, at best, just 1 or 2) beyond the limits of CMOS, then designing your devices around reversible computing principles is not optional – it’s an absolute necessity! Unfortunately, in my experience, there are far too few device physicists who seem to understand this basic fact.

Luke: If a researcher wanted to make progress toward reversible computing, where would you recommend they start looking? What are the most promising avenues of research at this point, do you think?

Mike: Well, as I mentioned earlier, I think that some fundamental breakthroughs in basic theory are still needed. We are still lacking a comprehensive theoretical model demonstrating how a realistic quantum-mechanical system composed of many interacting subsystems can be made to coast along a complexly-constrained, deterministic trajectory through configuration space with negligible entropy increase per operation. Reversible computing is somewhat easier than full-blown quantum computing, in that we don’t have to maintain delicate quantum superpositions of entangled states. Instead, we can presumably rely only on naturally stable (a.k.a. “classical”) states, or “pointer states.” But, accurately modeling the dynamical behavior of microscopic physical systems still requires a deep understanding of quantum mechanics; such an understanding is especially necessary for the design of nanoscale devices that could potentially serve as effective components of an efficient reversible computing system. If I were starting over in my career, I would begin by giving myself a comprehensive, top-of-the-line, world-class education in quantum mechanics, and then I’d re-develop the fundamental principles of reversible computing technology bottom-up, building on that foundation–with an emphasis on creating self-contained, energy-efficient, ballistic reversible computational mechanisms that are as simple as possible to design and manufacture. It is a big challenge, but I think that someone will show us the way eventually.

Luke: Thanks, Mike!
See slide 19 of Snider et al. (2013).See slides #89-93 of Frank (2006a).See slide #27 of Frank (2006b).See Plourde (2012); Ren (2012); Semenov (2012; Ustinov (2012).The post Mike Frank on reversible computing appeared first on Machine Intelligence Research Institute.

Mike Frank on reversible computing


Emil Vassev on Formal Verification

Dr. Emil Vassev received his M.Sc. in Computer Science (2005) and his Ph.D. in Computer Science (2008) from Concordia University, Montreal, Canada. Currently, he is a research fellow at Lero (the Irish Software Engineering Research Centre) at University of Limerick, Ireland where he is leading the Lero’s participation in the ASCENS FP7 project and the Lero’s joint project with ESA on Autonomous Software Systems Development Approaches. His research focuses on knowledge representation and awareness for self-adaptive systems. A part from the main research, Dr. Vassev’s research interests include engineering autonomic systems, distributed computing, formal methods, cyber-physical systems and software engineering. He has published two books and over 100 internationally peer-reviewed papers. As part of his collaboration with NASA, Vassev has been awarded one patent with another one pending.

Luke Muehlhauser: In “Swarm Technology at NASA: Building Resilient Systems,” you and your co-authors write that:
To increase the survivability of [remote exploration] missions, NASA [uses] principles and techniques that help such systems become more resilient…
…Practice has shown that traditional development methods can’t guarantee software reliability and prevent software failures. Moreover, software developed using formal methods tends to be more reliable.
When talking to AI scientists, I notice that there seem to be at least two “cultures” with regard to system safety. One culture emphasizes the limitations of systems that are amenable to (e.g.) formal methods, and advises that developers use traditional AI software development methods to build a functional system, and try to make it safe near the end of the process. The other culture tends to think that getting strong safety guarantees is generally only possible when a system is designed “from the ground up” with safety in mind. Most machine learning people I speak to seem to belong to the former culture, whereas e.g. Kathleen Fisher and other people working on safety-critical systems seem to belong to the latter culture.
Do you perceive these two cultures within AI? If so, does the second sentence I quoted from your paper above imply that you generally belong to the second culture?

Emil Vassev: Before going to the answer of your question, I need to clarify some of the basics of AI as I perceive it. Basically, AI depends on our ability to efficiently transfer knowledge to software-intensive systems. A computerized machine can be considered as one exhibiting AI when it has the basic capabilities to transfer data into context-relevant information and then that information into conclusions exhibiting knowledge. Going further, I can say that AI is only possible in the presence of artificial awareness, one by which we can transfer knowledge to machines. Artificial awareness entails much more than computerized knowledge, however. It must also incorporate means by which a computerized machine can perceive events and gather data about its external and internal worlds. Therefore, to exhibit awareness, intelligent systems must sense and analyze components as well as the environment in which they operate. Determining the state of each component and its status relative to performance standards, or service-level objectives, is therefore vital for an aware system. Such systems should be able to notice changes, understand their implications, and apply both pattern analysis and pattern recognition to determine normal and abnormal states. In other words, awareness is conceptually a product of representing, processing, and monitoring knowledge. Therefore, AI requires knowledge representation, which can be considered as a formal specification of the “brain” of an AI system. Moreover, to allow for learning, we must consider an open-world model of this “machine brain”.
Now, let’s to go back to your question. I think, both research “cultures” have their niche within AI. Both cultures lean towards the use of open-world modeling of the AI by using formal methods. The difference lies mainly in the importance of the safety requirements, which justifies both approaches. Note that AI is a sort of superior control mechanism that exclusively relies on the functionality of the system to both detect safety hazards and pursue safety procedures. Therefore, in all cases AI is limited by system functionality and systems designed “from the ground up with safety in mind” are presumably designed with explicit safety-related functionality, and thus, their AI is less limited when it comes to safety.
To answer your second question, yes, I consider myself as leaning towards the second “research culture”. For many NASA/ESA systems, safety is an especially important source of requirements. Requirements engineers can express safety requirements as a set of features and procedures that ensure predictable system performance under normal and abnormal conditions. Furthermore, AI engineers might rely on safety requirements to derive special self-* objectives controlling the consequences of unplanned events or accidents. Think about the self-* objectives as AI objectives driving the system in critical situations employing self-adaptive behavior. Safety standards might be a good source of safety requirements and consecutively on safety-related self-* objectives. Such self-* objectives may provide for fault-tolerance behavior, bounding failure probability, and adhering to proven practices and standards. Explicit safety requirements provide a key way to maintain safety-related knowledge within a machine brain of what is important for safety. In typical practice, safety-related AI requirements can be derived by a four-stage process:

Hazard identification – all the hazards exhibited by the system are identified. A hazard might be regarded as a condition – situation, event, etc., that may lead to an accident.
Hazard analysis – possible causes of the system’s hazards are explored and recorded. Essentially, this step identifies all processes, combinations of events, and sequences that can lead from a ‘normal’ or ‘safe’ state to an accident. Success in this step means that we now understand how the system can get to an accident.
Identifying safety capabilities – a key step is to identify the capabilities (functionality) the system needs to have in order to perform its goals and remain safe. It is very likely that some of the capabilities have been already identified by for the purpose of other self-* objectives.
Requirements derivation – once the set of hazards is known, and their causation is understood, engineers can derive safety requirements that either prevent the hazards occurring or mitigate the resulting accidents via self-* objectives.

Luke: As explained here, no system can be 100% safe, in part because its formal safety requirements might fail to capture a hazard that was missed, or inadequately modeled, by the system’s designers. What’s your intuitive sense of how much “extra” confidence in a system’s safety is typically gained by the application of formal methods? For example, you might say that space probe launched with lots of testing but no formal methods would have a 70% chance of completing its mission successfully, whereas a space probe which had been tested and formally verified (and hence, designed in a different way, so as to be amenable to formal verification) would have a 90% chance of completing its mission successfully.
Naturally, such estimates depend greatly on the details of the specific system, and which properties were proved by the formal verification process, but I’m hoping you can say something about your intuitive sense as to how much safety is gained in exchange for the extra work of designing a system so precisely that it can be formally verified.

Emil: Generally speaking, formal methods strive to build software right (and thus, reliable) by eliminating flaws, e.g., requirements flaws. Formal method tools allow comprehensive analysis of requirements and design and eventually near-to-complete exploration of system behavior, including fault conditions. However, good requirements formalization depends mainly on the analytical skills of the requirements engineers along with the proper use of the formal methods in hand. Hence, errors can be introduced when capturing or implementing safety requirements. This is maybe the main reason why, although efficient in terms of capacity of the dedicated analysis tools such as theorem provers and model checkers, formal methods actually do not eliminate the need of testing.
In regards with safety requirements, the application of formal methods can only add on safety. Even if we assume that proper testing can capture all the safety flaws that we may capture with formal verification, with proper use of formal methods we can always improve the quality of requirements and eventually derive more efficient test cases. Moreover, formal methods can be used to create formal specifications, which subsequently can be used for automatic test case generation. Hence, in exchange for the extra work put to formally specify the safety requirements of a system, you get not only the possibility to formally verify and validate these requirements, but also to more efficiently test their implementation.
So, to go back to your question, as you said 100% safety cannot be guaranteed, but when properly used, formal methods can significantly contribute to safety by not replacing, but complementing testing. The quantitative measure of how much safety can be gained with formal methods may be regarded in three aspects:

Formal verification and validation allows for early detection of safety flaws, i.e., before implementation.
High quality of safety requirements improves the design and implementation of these requirements.
Formally specified safety requirements assist in the derivation and generation of efficient test cases.

To be more specific, although it really depends on the complexity of the system in question, my intuition is that these three aspects complement each other and together they may help us build a system with up to 99% safety guarantee.

Luke: You wrote about the difficulty of “good requirements formalization” — that is, the challenge of translating our intuitive ideas about how we want the system to behave into a logical calculus.
What types of requirements are researchers currently able to formalize for use in formally verified systems? What types of intuitively desirable requirements are we as-yet unable to figure out how to formalize?

Emil: Contemporary formal verification techniques (e.g., model checking) rely on state-transition models where objects or entities are specified with states they can be in and associated with functions that are performed to change states or object characteristics. Therefore, basically every system property that can be measured or quantified, or qualified as a function can be formalized for the needs of formal verification. Usually, the traditional types of requirements – functional and non-functional (e.g., data requirements, quality requirements, time constraints, etc.), are used to provide a specific description of functions and characteristics that address the general purpose of the system. The formal verification techniques use the formal specification of such requirements to check desired safety and liveness properties. For example, to specify safety properties of a system, we need to formalize “nothing bad will happen to the system”, which can be done via the formalization of non-desirable system states along with the formalization of behavior that will never lead the system to these states.
Obviously, the formalization of well-defined properties (e.g., with proper states expressed via boundaries, data range, outputs, etc.) is a straightforward task. However, it is not that easy to formalize uncertainty, e.g., liveness properties (something good will eventually happen). Although, probabilistic theories such as fuzzy logic help us formalize “degrees of truth” and deal with approximate conclusions rather with exact ones, the verification tools for fuzzy control systems are not efficient due to the huge spate-explosion problem. Moreover, testing such systems is not efficient as well, simply because, statistical evidence for their correct behavior may be not enough. Hence, any property that requires a progressive evaluation (or partial satisfaction, e.g., soft goals) is difficult and often impossible to be formalized “for use in formally verified systems”.
Other properties that are “intuitively desirable” (especially by AI) but still cannot be formalized today are human behavior and principles, related to cultural differences, ethics, feelings, etc. The problem is that with the formal approaches today we cannot express, for example, emotional bias as a meaningful system state.

Luke: Let’s work with a concrete, if hypothetical, example. Suppose a self-driving car’s navigation software was designed with formal verification in mind, at least for many core components of the software. I presume we are very far from being able to define a formal requirement that matches the intuitive meaning of “Don’t ever be in a state such that the car has injured a pedestrian.”
So, what requirements could we formalize such that, if they were proven to be met by the car’s navigation software, would help us to feel increasingly confident that the car would never injure a pedestrian (while still allowing the car to engage in normal driving behavior)?

Emil: Again, this example should be regarded with the insight that “100 % safety is not possible”, especially when the system in question (e.g., a self-driving car) engages in interaction with a non-deterministic and open-world environment. What we should do though, to maximize the safety guarantee that “the car would never injure a pedestrian” is to determine all the critical situations involving the car itself in close proximity to pedestrians. Then we shall formalize these situations as system and environment states and formalize self-adaptive behavior (e.g., as self-* objectives) driving the car in such situations. For example, a situation could be defined as “all the car’s systems are in operational condition and the car is passing by a school”. To increase safety in this situation, we may formalize a self-adaptive behavior such as “automatically decrease the speed down to 20 mph when getting in close proximity to children or a school”.
Further, we need to specify situations involving close proximity to pedestrians (e.g., crossing pedestrians) and car states emphasizing damages or malfunction of the driving system, e.g., flat tires, malfunctioning stirring wheel, malfunctioning breaks, etc. For example, we may specify a self-adaptive behavior “automatically turn off the engine when the break system is malfunctioning and the car is getting in close proximity to pedestrians.”
Other important situations should involve severe weather conditions introducing hazards on the road, e.g., snow storm, ice, low visibility (formalized as environment states), and the car getting in close proximity to pedestrians. In such situations, formalized self-adaptive behavior should automatically enforce low speed, turning lights on, turning wipers on, etc.

Luke: To what degree, if any, can we demonstrate both deductive guarantees and probabilistic guarantees with today’s formal verification methods?
In a sense, many of the deductive proofs for safety properties in today’s formally verified systems are already “probabilistic” in the sense that the designers have some subjective uncertainty as to whether the formal specification accurately captures the intuitively desirable safety properties, and (less likely) whether there was an error in the proof somewhere.
But the question I’m trying to ask is about explicitly probabilistic guarantees within the proof itself. Though perhaps to do this, we’d first need a solution to the problem of how a formal system or logical calculus can have internally consistent probabilistic beliefs about logical sentences (a la Demski 2012 or Hutter et al. 2012).
If we could generalize formal specification and verification procedures to allow both for deductive guarantees and probabilistic guarantees, perhaps we could verify larger, more complex, and more diverse programs.

Emil: With deductive guarantees a formal verification actually provides true statements that demonstrate that desired safety properties are held. Such a verification process is deterministic and a complete proof is required to guarantee the correctness of safety properties. For example, such a proof can be equipped with deterministic rules and expressed in the classical first-order logic (or in high-order logic if we use Isabelle to run a deductive verification). On the other hand, with the probabilistic guarantees you can accept that a complete proof is not necessary and safety properties can be verified with some degree of uncertainty. Basically, the probabilistic guarantees can be regarded as a result of quantification of uncertainty in both the verification parameters and subsequent predictions. With the Bayesian methods, for example, we quantify our uncertainty as prior distribution of our beliefs we have in the values of certain properties. Moreover, we also embed likelihood in the properties formalization, i.e., how likely is it that we would observe a certain value in particular conditions. You may think about it as the likelihood of holding certain safety properties in specific situations. Then, the probabilistic guarantees assert in a natural way “likely” properties over the possibilities that we envision.
So, to answer your question, unfortunately, deductive guarantees can be provided only for simple safety properties, because their complete proof often unavoidably does not terminate. Although deductive verification may deal with infinite state systems, its automation is limited, which is mainly due to the decidability of the logical reasoning (first-order logic and its extensions such as high-order logic are not decidable, or rather semi-decidable). For example, if we go back to our example with the self-driving car, we may supply all the needed deterministic rules expressing our safety requirements (e.g., speed limit of 20 mph when passing by a school), but the complete proof eventually cannot be achieved, because although the desired conclusion follows from some of the premises, other premises may eventually lead to resolution refutation.
The probabilistic guarantees are not as complete as the deductive ones, but they may deal with more complex properties, e.g., where a larger number of states can be required. Of course, this tradeoff should be considered when evaluating the results of any probabilistic formal verification. So, if we go back to your question about how much confidence in system’s safety is gained with formal methods, probabilistic guarantees bring less confidence than deductive ones, but they may bring some extra confidence to safety properties that cannot be handled otherwise.
It is important to mention that abstraction is the most efficient solution to the state-explosion problem (and respectively, to the problem of deductive guarantees decidability). With abstraction the size of the state space is reduced by aggregating state transitions into coarser-grained state transitions. The technique effectively reduces the total amount of states to be considered but is likely to reduce the granularity of the system to a point where it no longer adequately represents that system. The problem is that although the abstract model (e.g, the formalization of safety properties) is relatively small it should also be precise enough to adequately represent the original system.
Therefore as you said, in order to obtain better results, we shall consider both verification approaches and eventually apply these together. For example, we may formalize with the presumption that both deductive and probabilistic guarantees can be obtained in a sort of compositional verification where we may apply both approaches to different safety properties, and eventually combine the results under the characteristics of global safety invariants. Such invariants can be classified as: goal invariants, behavior invariants, interaction invariants and resource invariants (see my co-authored paper “Verification of Adaptive Systems”).

Luke: Besides mixing probabilistic and deductive verification methods, what other paths forward do you see for improving on our current verification toolset? In particular, what important work in this area seems to be most neglected by funding and researcher hours?

Emil: Well, maybe the most popular technique for formal verification is model checking where the properties are expressed in a temporal logic and the system formalization is turned into a state machine. The model checking methods verify if the desired properties hold in all the reachable states of a system, which is basically a proof that properties will hold during the execution of that system. State explosion is the main issue model checking is facing today. This problem is getting even bigger when it comes to concurrent systems where the number of states is exponential to the number of concurrent processes. So, basically, model checking is an efficient and powerful verification method, but only when applied to finite, yet small state spaces.
Here, to answer your question about what can be done to improve the current verification toolset: on the one side we need to work on fully automated deductive verification based on decidable logics with both temporal and probabilistic features, and on the other side we need to work on improving the model checking ability to handle large state spaces (e.g., symbolic model-checking, probabilistic model checking, etc.).
Important work that seems neglected by the scientific community is the so-called stabilization science, which provides a common approach to studying system stability. In this approach, a system is linearized around its operating point to determine a small-signal linearized model of that operating point. The stability of the system is then determined using linear system stability analysis methods such as Routh-Hurwitz, Root Locus, Bode Plot, and Nyquist Criterion. We may use stabilization science to build small-signal linearized models for the different system components, anticipating that the linearized models of system components will yield a relatively small state space, enabling for their efficient verification (see again my co-authored paper “Verification of Adaptive Systems”). Then we may apply compositional verification techniques to produce an overall system-wide verification.
Other, not that well-developed verification techniques are those related to automatic test-case generation and simulation, which may reduce testing costs and improve the quality of testing. For example, test cases can be generated from a formal specification of a system built with a domain-specific formal language. If combined with code generation and analysis techniques for efficient test-case generation (e.g., change-impact analysis), automatic test-case generation might be used to efficiently test system behavior under simulated conditions (see my co-authored paper “Automated Test Case Generation of Self-Managing Policies for NASA Prototype Missions Developed with ASSL”).
Moreover, high-performance computing can be used for parallelizing simulations, which will allow multiple state space explorations to occur simultaneously, etc.

Luke: Thanks, Emil!
The post Emil Vassev on Formal Verification appeared first on Machine Intelligence Research Institute.

Emil Vassev on Formal Verification