Harley-ml/lesswrong huggingface dataset:
# New paper: Corrigibility with Utility Preservation

I am pleased to announce the availability of a long-format paper with new results on AGI safety: **Corrigibility with Utility Preservation**.  
  
You can get the paper at [https://arxiv.org/abs/1908.01695](https://arxiv.org/abs/1908.01695) , and in the related software repository at [https://github.com/kholtman/agisim](https://github.com/kholtman/agisim) .  
  
**Abstract**  
  
_Corrigibility is a safety property for artificially intelligent agents. A corrigible agent will not resist attempts by authorized parties to alter the goals and constraints that were encoded in the agent when it was first started. This paper shows how to construct a safety layer that adds corrigibility to arbitrarily advanced utility  
maximizing agents, including possible future agents with Artificial General Intelligence (AGI). The layer counter-acts the emergent incentive of advanced agents to resist such alteration.  
  
A detailed model for agents which can reason about preserving their utility function is developed, and used to prove that the corrigibility layer works as intended in a large set of non-hostile universes. The corrigible agents have an emergent incentive to protect key elements of their corrigibility layer. However, hostile universes may contain forces strong enough to break safety features. Some open problems related to graceful degradation when an agent is successfully attacked are identified.  
  
The results in this paper were obtained by concurrently developing an AGI agent simulator, an agent model, and proofs. The simulator is available under an open source license. The paper contains simulation results which illustrate the safety related properties of corrigible AGI agents in detail.  
_  
This post can be used for comments and questions.  
  
The paper contains several results and observations that do not rely on the heavy use of math, but other key results and discussions are quite mathematical. Feel to post questions and comments even if you have not read all the mathematical parts.  
  
As this is my first post on LessWrong, and my first paper on AGI safety, I feel I should say something to introduce myself. I have a Ph.D. in software design, but my professional life so far has been very diverse and multidisciplinary. Among other things I have been an experimental physicist, a standards developer and negotiator, an Internet privacy advocate, a wireless networking expert, and a systems architect in an industrial research lab. So I bring a wide range of tools and methodological traditions to the field. What made me interested in the field of AGI safety in particular is that it seems to have open problems where real progress can be made using mathematical techniques that I happen to like. I am currently on a sabbatical: basically this means that I decided to quit my day job, and to use my savings to work for a while on some interesting problems that are different from the interesting problems I worked on earlier.
# Project Proposal: Considerations for trading off capabilities and safety impacts of AI research

There seems to be some amount of consensus that people working on AI safety (at least within fairly mainstream ML / AI paradigms) shouldn't worry much about the effects of their projects on AI capabilities. New researchers might even try to push capabilities research forward, to build career capital. The best argument for this, IMO, is that someone else is probably going to do it if you don't, likely within the next 6 months (given the current pace of research).

I mostly agree with this view, but I do still think a bit about effects of my research on capabilities, and think others should as well. Being concerned about advancing capabilities has, in the past, moved me away from pursuing ambitious capabilities projects which might have been very good for my career if they paid off, but I always saw someone else do the thing I was considering soon afterwards anyways...

**But as far as I know, nobody has tried to evaluate this question thoroughly and systematically.** This is concerning, because it seems like current attitudes could plausibly be a result of motivated reasoning (i.e. "I want to keep doing my research, and probably would do so even if I saw a compelling case against it") and groupthink ("nobody else is worrying about this"). I'm not sure it's really tractable, but I think it could be worth ~1-4 people spending a bit of time (possibly up to ~6-24 months, if it ends up looking tractable after some initial thought/investigation) on trying to do a fairly comprehensive treatment of this question.

The main deliverables could be practical guidelines for AI safety researchers, e.g.:

*   Figuring out when it makes sense to be concerned about advancing AI capabilities via ones' research.
*   How to decide how significant those concerns are, and whether they should preclude working on that line of research or research project, or change the publication model for them.

The project could intersect with current "dual-use" considerations (e.g. RE GPT-2).

(also worth mentioning): I know MIRI now has secret research, and I think they have a reasonable case for that, since they aren't in the mainstream paradigms. I do think it would be good for them to have a "hit publication" within the ML community, and might be worth pushing some out-of-the-box ideas which might advance capabilities. The reason is that MIRI has very little credibility, or even name recognition in the ML community, ATM, and I think it would be a big deal in terms of "perception of AI safety concerns within the ML community" if that changed. And I think the ML communities perceptions are important, because the ML community's attitude seems of critical importance for getting good Xrisk reduction policies in place (IIRC, I talked to someone at MIRI who disagreed with that perspective).

The idea to write this post came out of discussion with Joe Collman.
# Help forecast study replication in this social science prediction market

_This is a linkpost for: [https://www.replicationmarkets.com/](https://www.replicationmarkets.com/)_

From their sign-up page:

> Help the Replication Markets team spot reliable science, and win $$!

> From mid-2019 to mid-2020, we will run prediction markets on 3,000 recently-published social and behavioral science claims. We want you to forecast the outcomes of possible replication trials. On average, only about 50% of published social science results replicate, but it’s not a coin toss: four previous markets have been about 75% accurate. We think we can beat 80%, while forecasting 10-100x as many claims.

> Join us to improve social and behavioral science, try new kinds of markets and surveys, and earn $$ prizes. (Over $100,000 in total prizes, distributed via Google Pay, among a target pool of 500 forecasters based on accuracy and contributions.)

> Please look around, see the Replication Markets Home Page for more details, or click “[Sign Up](https://predict.replicationmarkets.com/main/#!/users/register)” to join.
# Which of these five AI alignment research projects ideas are no good?

I'll post five AI alignment research project ideas as comments. It would be
great if you could approval-vote on them by using upvotes. Ie. when you think
the project idea isn't good, you leave the comment as is; otherwise you give it
a single upvote.

The project ideas follow this format (cf. [The Craft of
Research](https://press.uchicago.edu/ucp/books/book/chicago/C/bo23521678.html)):

    I'm studying <topic>,
        because I want to <question that guides the search>,
            in order to help my reader understand <more significant
            question that would be informed by an answer to the
            previous question>.

The project ideas are fixed-width in order to preserve the indentation. If they get formatted strangely, you might be able to fix it by increasing the width of your browser window or zooming out.
# Why do humans not have built-in neural i/o channels?

Communication between organisms of the same species is often beneficial, for a variety of reasons: sharing information, signalling, bonding, etc. Yet currently the most advanced form of communication to have evolved, human language, is still very low bandwidth compared with the amount of mental processing our brains do.

It seems conceivable that our nervous systems might have evolved ways to directly (temporarily) interface with each other and exchange a large amount of information. For example, retractable bundles of neurons that are specialised at quickly forming connections with their counterpart neurons in conspecifics.

1\. What is the main reason that this has not happened in any large animals? If evolution had continued without humans "taking off", we would have eventually seen such neural links in some animal species?

2\. What's the closest thing to this we see in any species?
# What is the state of the ego depletion field?

It's been almost a half decade since the replication crisis in psychology broke, and one of the major casualties was the subfield of ego depletion. (I belive [this](https://journals.sagepub.com/doi/full/10.1177/1745691616652873) was the initial large scale meta-analysis that found no effect. And [here](http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.html) is a popular article on the failure to replicate.)

Does anyone have a good summary of the state of that research program? Is there any part of it that held up? What can we conclusively say about self control, willpower, depletion of mental resources, etc?
# AI Safety Reading Group

If you are interested in AI Safety, come visit the AI Safety Reading Group.  
  
The AI Safety reading group meets on Skype Wednesdays at 18:45 UTC, discussing new and old articles on different aspects of AI Safety. We start with a presentation round, then a summary of the article is presented, followed by discussion both on the article and in general.  
  
Sometimes we have guests. On Wednesday the 14th, Stuart Armstrong will be giving a presentation on his research agenda in the reading group:[  
https://www.alignmentforum.org/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-into](
https://www.alignmentforum.org/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-into

Join)

Join us by Skype, by adding ‘soeren.elverlin’.  
  
Previous guests include Eric Drexler, Rohin Shah, Matthijs Maas, Scott Garrabrant, Robin Hanson, Roman Yampolskiy, Vadim Kosoy, Abram Demski and Paul Christiano. A full list of articles read can be found at [https://aisafety.com/reading-group/](https://aisafety.com/reading-group/)
# What explanatory power does Kahneman's System 2 possess?

In the 70s and 80s, Kahneman and Tversky did a bunch of pioneering research on heuristics and biases in human thought. Then, in _Thinking Fast and Slow,_ Kahneman divided human cognition into System 1 and System 2 - basically, System 1 applies quick heuristics which are prone to biases, and System 2 does the slow, effortful thinking.

But what does System 2 actually add to the theory in terms of explanatory power? Consider an alternative version of _Thinking Fast and Slow_ in which Kahneman wrote something like "Here are the conditions in which humans use this mode of reasoning I'm calling System 1, which is fast and approximate and effortless and uses heuristics and demonstrates biases which can be detected in certain ways. The rest of the time, I have no idea what's going on, except that it doesn't display the traits that would qualify it as System 1 inference." In what ways would this be less informative than his actual claims?
# How can you use music to boost learning?

I often find that I am able to appreciate the beauty of a subject more while listening to music (especially instrumental music). Hearing the notes while I think about the topic helps creates a lot of subconscious connections with the material, solidifying what I am learning as a distinct set of memories. In general I think that associations with powerful sensory experience is just a good way to remember things and learn.

However, I have also heard that listening to music can distract you when you are trying to do deep work. Apparently, there is some research on this, but I have barely scratched the surface of the literature, and I wouldn't know where to start.

Is there an optimal way to use music to learn? Should I employ certain strategies, like putting on the music only after I've read something, so that I can think about what I just read while music floats through my consciousness?
# Problems in AI Alignment that philosophers could potentially contribute to

(This was originally a comment that I wrote as a follow up to [my question](https://ea.greaterwrong.com/posts/oPGJrqohDqT8GZieA/ask-me-anything/comment/cL3KFfJwtHZKeBPGH) for William MacAskill's AMA. I'm moving it since it's perhaps more on-topic here.) 

It occurs to me that another reason for the lack of engagement by people with philosophy backgrounds may be that philosophers aren't aware of the many philosophical problems in AI alignment that they could potentially contribute to. So here's a list of philosophical problems that have come up just in my own thinking about AI alignment.

* Decision theory for AI / AI designers
  * How to resolve standard debates in decision theory?
  * Logical counterfactuals
  * Open source game theory
  * Acausal game theory / reasoning about distant superintelligences
* Infinite/multiversal/astronomical ethics
  * Should we (or our AI) care much more about a universe that is capable of doing a lot more computations?
  * What kinds of (e.g. spatial-temporal) discounting is necessary and/or desirable?
* Fair distribution of benefits
  * How should benefits from AGI be distributed? 
  * For example, would it be fair to distribute it equally over all humans who currently exist, or according to how much AI services they can afford to buy?
  * What about people who existed or will exist at other times and in other places or universes?
* Need for "metaphilosophical paternalism"?
  * However we distribute the benefits, if we let the beneficiaries decide what to do with their windfall using their own philosophical faculties, is that likely to lead to a good outcome?
* Metaphilosophy
  * What is the nature of philosophy?
  * What constitutes correct philosophical reasoning?
  * How to specify this into an AI design?
* Philosophical forecasting
  * How are various AI technologies and AI safety proposals likely to affect future philosophical progress (relative to other kinds of progress)?
* Preference aggregation between AIs and between users
  * How should two AIs that want to merge with each other aggregate their preferences?
  * How should an AI aggregate preferences between its users?
* Normativity for AI / AI designers
  * What is the nature of normativity? Do we need to make sure an AGI has a sufficient understanding of this?
* Metaethical policing
  * What are the implicit metaethical assumptions in a given AI alignment proposal (in case the authors didn't spell them out)? 
  * What are the implications of an AI design or alignment proposal under different metaethical assumptions?
  * Encouraging designs that make minimal metaethical assumptions or is likely to lead to good outcomes regardless of which metaethical theory turns out to be true.
  * (Nowadays AI alignment researchers seem to be generally good about not placing too much confidence in their own moral theories, but the same can't always be said to be true with regard to their metaethical ideas.)
# Inspection Paradox as a Driver of Group Separation

> _Have you ever noticed that anybody driving slower than you is an idiot, and anyone going faster than you is a maniac?_ \-\- George Carlin

The [Inspection Paradox](http://allendowney.blogspot.com/2015/08/the-inspection-paradox-is-everywhere.html), where the reported results are heavily observer-dependent, has been mentioned here a couple of times before:

[https://www.lesswrong.com/posts/tqEAiScpz7g2xLHtE/the-inspection-paradox-is-everywhere](https://www.lesswrong.com/posts/tqEAiScpz7g2xLHtE/the-inspection-paradox-is-everywhere)

[https://www.lesswrong.com/posts/HW2fbbGM8B6y7pkDb/the-just-world-hypothesis#8r2uaA6Mb25k2vF9Z](https://www.lesswrong.com/posts/HW2fbbGM8B6y7pkDb/the-just-world-hypothesis#8r2uaA6Mb25k2vF9Z)

An example of it that is familiar to everyone is that, when driving with an average speed, you see all other cars separating into two categories: slow drivers and fast drivers, because you naturally encounter more cars who are faster or slower than you are, and none that move at the same speed. So a normal distribution of speeds becomes bimodal, from something like this:

![](https://i.gyazo.com/4e1051a8dac03ff3609fff97f138a7a5.png)

to something like this:

![](https://i.gyazo.com/a91b5a42c4f396a01dcc4d003cc277e9.png)

Another familiar example, from the [newer post by the original author](https://medium.com/@downey_78309/the-inspection-paradox-is-everywhere-2ef1c2e9d709), is that for an average Facebook user,

> Your friends have more Facebook friends than you do.

Because, naturally, members with more connections are more likely to have a connection with you, among others.

(For the record, the inspection paradox can be classified as subset of the [sampling bias](https://en.wikipedia.org/wiki/Sampling_bias), which, in turn, is a form of the oft discussed [selection bias](https://en.wikipedia.org/wiki/Selection_bias).)

But back to the apparent multi-modality. You encounter is whenever unusual events have a higher availability, sometimes properly measured like in the average driver case above, and sometimes perceived, like in the [availability heuristic](https://en.wikipedia.org/wiki/Availability_heuristic). This doesn't have to be about the frequency of the observations, it could be about their emotional impact. If the more "out there" the observations are, the more they affect you, then their distribution would appear bimodal, and you might instinctively recoil from them, and seek the comfort of the in-group. In this case the inspection-induced multi-modality may turn into an actual one, a case of perception becoming reality. (I had attempted to model it numerically some months ago, in two [companion](https://www.lesswrong.com/posts/JigXyiFaiAZ6zY9d6/quantitative-philosophy-why-simulate-ideas-numerically) [posts](https://www.lesswrong.com/posts/7uzTGqerKKeML7M2K/a-numerical-model-of-view-clusters-results), sadly not written well enough to attract much interest, but showing that the effect described may well be real.)
# "Can We Survive Technology" by von Neumann

> "The great globe itself" is in a rapidly maturing crisis —a crisis attributable to the fact that the environment in which technological progress must occur has become both undersized and underorganized. To define the crisis with any accuracy, and to explore possibilities of dealing with it, we must not only look at relevant facts, but also engage in some speculation. The process will illuminate some potential technological developments of the next quarter-century.

Added: see orthonormal's comment below for a summary.
# Do We Change Our Minds Less Often Than We Think?

In "[We Change Our Minds Less Often Than We Think](https://www.lesswrong.com/posts/buixYfcXBah9hbSNZ/we-change-our-minds-less-often-than-we-think)", Eliezer quotes a study:

> Over the past few years, we have discreetly approached colleagues faced with a choice between job offers, and asked them to estimate the probability that they will choose one job over another. The average confidence in the predicted choice was a modest 66%, but only 1 of the 24 respondents chose the option to which he or she initially assigned a lower probability, yielding an overall accuracy rate of 96%.

> —Dale Griffin and Amos Tversky

Eliezer then notes that this radically changed the way he thought:

> When I first read the words above—on August 1st, 2003, at around 3 o’clock in the afternoon—it changed the way I thought. I realized that o_nce I could guess what my answer would be—o_nce I could assign a higher probability to deciding one way than other—then I had, in all probability, already decided. We change our minds less often than we think. And most of the time we become able to guess what our answer will be within half a second of hearing the question.

> \[...\]

> But we change our minds less often—m_uch_ less often—than we think.

But a) this seems like it's pre-replication crisis, b) regardless, a sample size of 24 is not nearly high enough for me to be very confident in this.

"How often people change their mind" seems like a fairly important question. Anyone know of further work in similar space here? Ideally asking the question from a few different angles.
# Negative "eeny meeny miny moe"

As a kid, I learned the rhyme as:

> Eeny, meeny, miny, moe,

> Catch a tiger by the toe.

> If he hollers, let him go,

> Out goes Y, O, U!

Since kids can't predict where it will end, and adults are not supposed to try, it's a reasonably fair way of drawing lots.

At times I've heard versions where the selected person wins instead of loses, and while with two kids it doesn't matter, with three or more it matters a lot!

Let's model each kid having a choice at each stage between "accept" and "protest". While protesting probably doesn't work, if enough of you protest it might. If you do the positive version, where the selected kid wins, the winner accepts but the others may choose to protest. This isn't good: everyone has reason to protest except the single winner.

On the other hand, with the negative version, where one kid is eliminated at once, it's the other way around. When the first kid is eliminated they may protest, but the other kids all accept because then they retain their chance to win. With each successive round the dynamic is the same, plus the already-eliminated kids all choose accept out of a desire for fairness. Even with the last elimination there's still only one person choosing protest.

The iterative process is `O(n)` instead of `O(1)`, but it also works much better because it keeps a majority for "accept" at each stage.

(If you have a very large group of kids, then I could imagine a `O(log(n))` version being worth the added complexity. Divide the kids into three groups, and do negative eeny meeny miny moe on the groups. A third of the kids may protest, but you've still got two thirds accepting. Then redivide those remaining two thirds into three groups, and keep going.)

*cross-posted from https://www.jefftk.com/p/negative-eeny-meeny-miny-moe*
# Has Moore's Law actually slowed down?

Moore's Law has been notorious for spurring a bunch of separate observations that are all covered under the umbrella of "Moore's law." But as far as I can tell, the real Moore's law is a fairly narrow prediction, which is that the transistor count on CPU chips will double approximately every two years.

Many people have told me that in recent years Moore's law has slowed down. Some have even told me they think it's stopped entirely. For example, the AI and compute [article](https://openai.com/blog/ai-and-compute/) from OpenAI uses the past tense when talking about Moore's Law, "by comparison, Moore’s Law had an 18 month doubling period" (note that Wikipedia says, "The period is often quoted as 18 months because of a prediction by Intel executive David House").

Yet when I google Moore's law, I find [this chart](https://www.karlrupp.net/2018/02/42-years-of-microprocessor-trend-data/) from Karl Rupp's website,

![](https://www.karlrupp.net/wp-content/uploads/2018/02/42-years-processor-trend.png)

To me this looks very stable. And even though single thread performance has stopped improving by much, I can't really see how it's justified to use Moore's law in the past tense just yet. Am I missing something?
# What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause?

ETA: I'll be adding things to the list that I think belong there.

I'm assuming a high level of credence in classic utilitarianism, and that AI-Xrisk is significant (e.g. roughly >10%), and timelines are not long (e.g. >50% ASI in <100years). ETA: For the purpose of this list, I don't care about questioning those assumptions.

Here's my current list (off the top of my head):

*   not your comparitive advantage
*   consider other Xrisks more threatening (top contenders: bio / nuclear)
*   infinite ethics (and maybe other fundamental ethical questions, e.g. to do with moral uncertainty)
*   S-risks
*   simulation hypothesis
*   ETA: AI has high moral value in expectation / by default
*   ETA: low tractability (either at present or in general)
*   ETA: Doomsday Argument as overwhelming evidence against futures with large number of minds

Also, does anyone want to say why they think none of these should change the picture? Or point to a good reference discussing this question? (etc.)
# What authors consistently give accurate pictures of complex topics they discuss?

When I read nonfiction, I often feel like it's a waste of time because I'm not very confident the author is portraying things accurately/non-misleadingly, and don't often have time or expertise to check (though I'd potentially be in doing this on 1/10 of books I read, or similar, to contribute to a group effort to source reliably books).

I am looking for people who write nonfiction books about complicated topics on which expert views vary (social sciences, theories of why/how important historical events happened, nutrition, overall takes on complex scientific questions like the expected economic effects of climate change) that pass spot checks, seem to weigh evidence rationally, seem focused on truth-seeking over promoting their pet theory/agenda/sensationalistic conclusion, pay more attention to the most important parts of the issue than the minutia, etc. I am guessing this trait generalizes between books written by the same author, but would be interested in cases in which this doesn't seem to hold.

I don't want suggestions for people in the rationalist community or very adjacent to it.
# GPT-2: 6-Month Follow-Up

Linkpost for [GPT-2 6 Month Follow-Up](https://openai.com/blog/gpt-2-6-month-follow-up/).

Some highlights:
- 700+ M parameter model is being released
- Several other groups have reproduced similar models
- In detecting synthesized text, "current ML-based methods only achieve low to mid–90s accuracy"
# Paradoxical Advice Thread

_(Edited to be more clear)_

One can find plenty of witty advice catch phrases that appear to be in direct conflict with each other: "Early bird gets the work, but the second mouse gets the cheese" (from a comment below)

You can also find this with things that are more detailed frames than cliche advice. One that I've been mulling over is "decision makers need skin in the game to not be act stupid" and "asymmetric justice (people being punished for messing up) incentives inaction."

The "quality" of any advice varies wildly with what you are trying to do and what your specific context is. It's not going to be the case that the "best" thing for everyone to do is either follow the mandate of "have skin in the game" or "avoid asymmetric justice". In fact, you might be thinking about ways that those to ideas aren't even _really_ at odds with each other. Yet my personal experience was hearing about skin in the game, going "that sounds obviously important", hearing about asymmetric justice and going "this sounds obviously important", and later in a conversation with a friend going, "Wait a second, my low resolution understanding of these ideas puts them in direct odds with each other."

I've learned a bit from hunting for pairs of advice/principles that both sound like good ideas to me, but at first glance seem to be at odds. _Then_ exploring "why do they feel like they conflict?" Sometimes I find that they don't actually advise completely different approaches. Sometimes I find that each piece of advice is paying attention to a different dimension of trade off, and that conceivably I could do well by paying attention to each dimension separately.

I'm inviting people to use this post to document paradoxes you find in your own thinking. Explore where the sense of conflict comes from, what there is to learn from each side of the, what a synthesis might be, or when one might apply more than another.

Don't feel like you have to solve the whole thing, just try and unravel the problem space a bit.
# Announcement: Writing Day Today (Thursday)

The MIRI Summer Fellows Program is having a writing day, where the participants are given a whole day to write whatever LessWrong / AI Alignment Forum posts that they like. This is to practice the skills needed for [forum participation as a research strategy](https://www.lesswrong.com/posts/rBkZvbGDQZhEymReM/forum-participation-as-a-research-strategy).

On an average day LessWrong gets about 5-6 posts, but last year [this generated 28 posts](https://www.lesswrong.com/s/3qa3jAE9sqFqH9okL). It's likely this will happen again, starting mid-to-late afternoon PDT.

It's pretty overwhelming to try to read-and-comment on 28 posts in a day, so we're gonna make sure this isn't the only chance you'll have to interact with these posts. In the coming weeks I'll post roundups, highlighting 3-5 of the posts, giving them a second airing for comments and discussion.
# Time Travel, AI and Transparent Newcomb

_Epistemic status: has "time travel" in the title._

Let's suppose, for the duration of this post, that the local physics of our universe allows for time travel. The obvious question is: how are paradoxes prevented?

We may not have any idea how paradoxes are prevented, but presumably there must be _some_ prevention mechanism. So, in a purely Bayesian sense, we can condition on paradoxes somehow not happening, and then ask what becomes more or less likely. In general, anything which would make a time machine more likely to be built should become less likely, and anything which would prevent a time machine being built should become more likely.

In other words: if we're trying to do something which would make time machines more likely to be built, this argument says that we should expect things to mysteriously go wrong.

For instance, let's say we're trying to build some kind of powerful optimization process which might find time machines instrumentally useful for some reason. To the extent that such a process is likely to build time machines and induce paradoxes, we would expect things to mysteriously go wrong when trying to build the optimizer in the first place.

On the flip side: we could commit to designing our powerful optimization process so that it not only avoids building time machines, but also actively prevents time machines from being built. Then the mysterious force should work in our favor: we would expect things to mysteriously go well. We don't need time-travel-prevention to be the optimization process' sole objective here, it just needs to make time machines sufficiently less likely to get an overall drop in the probability of paradox.
# Does Agent-like Behavior Imply Agent-like Architecture?

This is not a well-specified question. I don't know what "agent-like behavior" or "agent-like architecture" should mean. Perhaps the question should be "Can you define the fuzzy terms such that 'Agent-like behavior implies agent-like architecture' is true, useful, and in the spirit of the original question." I mostly think the answer is no, but it seems like it would be really useful to know if true, and the process of trying to make this true might help us triangulate what we should mean by agent-like behavior and agent-like architecture.

Now I'll say some more to try to communicate the spirit of the original question. First a giant look-up table is not (directly) a counterexample. This is because it might be that the only way to produce an agent-like GLUT is to use agent-like architecture to search for it. Similarly a program that outputs all possible GLUTs is also not a counterexample because you might have to use your agent-like architecture to point at the specific counterexample. A longer version of the conjecture is "If you see a program implements agent-like behavior, there must some agent-like architecture in the program itself, in the causal history of the program, or in the process that brought your attention to the program." The pseudo-theorem I want is similar to the claim that correlation really does imply causation or the [good regulator theorem](https://en.wikipedia.org/wiki/Good_regulator).

One way of defining agent-like behavior as that which can only be produced by an agent-like architecture. This makes the theorem trivial, and the challenge is making the theorem non-vacuous. In this light, the question is something like "Is there some nonempty class of architectures that can reasonably be described as a subclass of 'agent-like' such that the class can be equivalently specified either functionally or syntactically?" This looks like it might conflict with the spirit of [Rice's theorem](https://en.wikipedia.org/wiki/Rice%27s_theorem), but I think making it probabilistic and referring to the entire causal history of the algorithm might give it a chance of working.

One possible way of defining agent-like architecture is something like "Has a world model and a goal, and searches over possible outputs to find one such that the model believes that output leads to the goal." Many words in this will have to be defined further. World model might be something that has high logical mutual information with the environment. It might be hard to define search generally enough to include everything that counts as search. There also might be completely different ways to define agent-like architecture. Do whatever makes the theorem true.
# Redefining Fast Takeoff

_This post is a result of numerous discussions with other participants and organizers of the MIRI Summer Fellows Program 2019. It describes ideas that are likely already known by many researchers. However, given how often disagreements about slow/fast takeoffs come up, I believe there is significant value in making them common knowledge._
_______________ 


# Takeoff speed & why it matters
In Superintelligence, Nick Bostrom distinguishes between slow, medium, and fast AI takeoff scenarios (where the takeoff speed is measured by how much real-world time passes between the milestones of human-level AI (HLAI) and superintelligent AI (SAI)).
He argues that slow takeoff should be reasonably safe since the humanity would have sufficient time coordinate and solve the AI alignment problem, while fast takeoff would be particularly dangerous since we wouldn't be able to react to what the AI does.


# Real-world time is not what matters
In many scenarios, the real-time takeoff speed indeed strongly correlates with our ability to influence the outcome. However, we can also imagine many scenarios where this is not the case.
As an example, suppose we obtain HLAI by simulating humans in virtual environments, and that this procedure additionally fully preserves the simulated humans' alignment with humanity. Since this effectively increases the speed at which humanity operates, we might get a "fully controlled takeoff" even if the transition from HLAI to SAI[^1] only takes a few days of real-world time.
More generally, if our path to HLAI also increases the effectivity of humanity's efforts, the "effective time" we get between HLAI and SAI will scale accordingly. For example, this _might_ be the case if we go the way of Iterated Distillation and Amplification or Comprehensive AI Services. Less controversially, suppose we automate most of the current programming tasks and increase the re-usability of code, such that every computer scientist becomes 100-times as effective as they are now.


# Conclusion: measure useful work
Given these examples, **I think we should measure takeoff speeds not in real-world time, but rather in** (some operationalization of) **the work-towards-AI-alignment that humanity will be able to do between HLAI and SAI**. Anecdotal examples of such measures might include "integral of the human-originating GDP between HLAI and SAI" or "number of AI safety papers published between HLAI and SAI".
I believe that finding a non-anecdotal operationalization would benefit many AI policy/strategy discussions.


[^1]: Recall that Bostrom distinguishes between speed, collective, and quality superintelligence. Arguably, being able to simulate humans (with enough compute) already constitutes a speed superintelligence. However, I don't think this diminishes the overall point of the post.
# A basic probability question

From the logical induction paper [https://intelligence.org/files/LogicalInductionAbridged.pdf:](https://intelligence.org/files/LogicalInductionAbridged.pdf:)

Let φ stand for the claim that the 87,653rd digit of π is a 7. If this claim is true, then (1 + 1 = 2) ⇒ φ. I don't understand the second sentence at all, any help appreciated.
# Is there a simple parameter that controls human working memory capacity, which has been set tragically low?

Here's the kind of thing I mean by "human working memory capacity" being "set tragically low": A 100-word sentence introducing a new concept is often annoyingly hard to wrap one's head around, compared to a longer and more "gentle" explanation.

The concept of working memory seems like a useful reduction of part of what makes intelligence work:

Most of us here think human intelligence can one day be increased a lot, because there doesn't seem to be a fundamental limit to intelligence located anywhere near human-level, but that's a non-constructive reductio type of argument.

On the other hand, it seems like there's an obvious constructive argument to why working memory should be able to be increased: Take whatever set of structures encode the current short-term brain's working memory contents, and analyze how big they are, like how many copies of some lower-level storage unit they have, and imagine architecting the brain to have more such copies of those systems encoding more memory contents.

Getting back to my question...

We all wish we were smarter, but we suspect that there are no easily-tweakable parameters that would yield a big increase in intelligence, or natural selection would have already tweaked them.

But doesn't it feel like working memory is an easily-tweakable parameter, and natural selection just didn't tweak it because it was good enough relative to the rest of the brain and the needs of the species, but it could actually be tweaked pretty easily in a couple thousand generations if there were the selection pressure to do it?

The reason it might make sense to say our small working memories are particularly **tragic** is if there is, as I suspect, a smaller counterfactual distance to them being larger and much more useful for modern life.
# LessLong Launch Party

Link to Facebook event: [https://www.facebook.com/events/714774672318252/](https://www.facebook.com/events/714774672318252/)

**WOO, A PARTY!**

The LessWrong team wants to celebrate the launch of Shortform (description below) so we're going to host a big, long party that is all things short.

Come dressed in shorts, short-sleeved shirts, and short other clothes (but, like, not \_too\_ short). Have brief conversations (but also some long ones). Drink small but intense drinks.

Speak off-the-cuff, share those things that are a little rough. That's the Shortform spirit.

**WHAT'S SHORTFORM/LESSLONG?**

LessWrong recently launched [Shortform](http://www.lesswrong.com/shortform)! (aka [www.lesslong.com](http://www.lesslong.com/))

Shortform is a new section of the LessWrong site for freely making short posts \[1\] about whatever's on your mind. Incomplete, exploratory, formative, rambly, rough, boring, draft-stage, and unedited thoughts are all welcome on Shortform.

The idea is to create a place for people 1) to record and share their ideas even before they hit final stages of polishing. Too many good ideas never it to that stage, 2) have somewhere to put short bits of interesting thoughts which aren't yet full-blown ideas.

Read more about the philosophy of Shortform here: [https://www.lesswrong.com/posts/tFwNi24kXeZ2BzvZz/shortform-beta-launch](https://www.lesswrong.com/posts/tFwNi24kXeZ2BzvZz/shortform-beta-launch
or)

\[1\] Or posts which took a _relatively_ short amount of time to write despite being a normal length.
# Is LW making progress?

Some people have an intuition that with free exchange of ideas, the best ones will eventually come out on top. I'm less optimistic, so I ask if that's really happening.

The alternative would be to have the same people talking about the same problems without accumulating anything. People would still make updates, but some would update in opposite directions, leaving the total distribution of ideas in the community largely unchanged. There would be occasional great ideas, but they would soon get buried in the archives, without leaving much of an impact. I have some hope we're not living in that world and I want to see evidence to justify it.

One way to look at the question is to consider the sequences. They are 10 years old now. Are they in some way outdated? Are there some new and crucial ideas that they are missing?
# Under a week left to win $1,000! By  questioning Oracle AIs.

That clickbait-y title is, for once, perfectly accurate: you have until the 31st of August to submit an entry to the [Oracle design contest](https://www.lesswrong.com/posts/cSzaxcmeYW6z7cgtc/contest-usd1-000-for-good-questions-to-ask-to-an-oracle-ai), either a counterfactual or a low bandwidth Oracle.

(A note: some of the submissions were done by people who clearly didn't understand how these types of Oracles worked; please check these details before submitting. In particular, any question you ask also has to contain some automated way of checking the Oracle's answer).
# How Can People Evaluate Complex Questions Consistently?

I'm doing a project on how humans can evaluate messy problems and come up with consistent answers (consistent with both themselves over time and with other people), and what the trade off is with accuracy. This isn't a single unified field, so I need to go poking for bits of it in lots of places. Where would people suggest I look? I'm especially interested in information on consistently evaluating novel questions that don't have enough data to make 
statistical models ("When will [country] develop the nuclear bomb?") as opposed to questions for which we have enough data that we're pretty much just looking for similarities ("Does this biopsy reveal cancer?").

An incomplete list of places I have looked or plan on looking at:
* interrater reliability
* test-retest reliability
* educational rubrics (for both student and teacher evaluations)
* medical decision making/standard of care
* Daniel Kahneman's work
* Philip Tetlock's work
* The Handbook of Inter-Rater Reliability
# Toy model piece #3: close and distant situations

I'll build on my model of [the previous post](https://www.lesswrong.com/posts/gzwM9edW4fXPwJuAW/toy-model-piece-2-combining-short-and-long-range-partial), and consider close and distant situation.

Recall that each world in $\mathcal{W}$ is defined by $0\leq n \leq 100$, which is the number of people smiling in it. I'll extend the model by adding another Boolean variable $b$, which determines, say, whether the world contains chocolate ($b=0$) or art ($b=1$). So worlds can be described by the pair $(n, b)$.

The default situation - the one if the AI does nothing - is $(45,1)$, say. So $45$ smiling people in a world of art.

Then let's introduce two new partial preferences/pre-orders:

* $P_3 = \{(n,1) \leq (m,1) \mid 40\leq n \leq m \leq 50\}$.
* $P_4 = \{(n,b) \leq (m,1) \mid 40 \leq m \leq 50\}$.

So $P_3$ say that, within a range of the default world (art, and the number of smiling people not being within $5$ people of $45$), the more smiling people, the better. While $P_4$ says that worlds in this range are better than worlds outside them.

These result in the following utility functions:

* $U_3(n,1) = 2n - 90$ if $40\leq n\leq 50$, $U_3(n,b)=0$ otherwise.
* $U_4(n,1) = 1$ if $40\leq n\leq 50$, $U_4(n,b)=-1$ otherwise.

After [normalisation](https://www.lesswrong.com/posts/gzwM9edW4fXPwJuAW/toy-model-piece-2-combining-short-and-long-range-partial), these become:

* $\widehat{U}_3(n,1) = n/5 - 9$ if $40\leq n\leq 50$, $U_3(n,b)=0$ otherwise.
* $\widehat{U}_4(n,1) = 1$ if $40\leq n\leq 50$, $U_4(n,b)=-11/191$ otherwise.

Again, I've felt free to translate $\widehat{U}_4$ to improve the clarity of the normalised version.

If we plot $\widehat{U}_3$, we get the following:

![](https://www.dropbox.com/s/7cul7ui2st6p8rn/close_pref.png?raw=1)

Here I've slightly offset the $b=1$ (purple) from the $b=0$ (blue) worlds, for clarity of exposition, though they would of course be on top of each other.

Note that $\widehat{U}_3$ does not in itself distant situations, as [this post](https://www.lesswrong.com/posts/PX8BB7Rqw7HedrSJd/by-default-avoid-ambiguous-distant-situations) recommends doing. Five close worlds are ranked about the distant worlds; but, conversely, five are close worlds are ranked below them.

To avoid distant situations, we need to add in $\widehat{U}_4$, which explicitly punishes distant worlds, and hence plot $\widehat{U}_3+\widehat{U}_4$:

![](https://www.dropbox.com/s/6s2lfpnrtrdxiwc/close_distant_pref.png?raw=1)

This is much more like it! All the close worlds are now all ranked above the more distant ones.

But this is a close-run thing: the difference between the worse close world and the distant worlds is small. So, in general, when penalising distant worlds, we have to do this with some care, and maybe use a different standard of normalisation (since the downgrading of distant worlds is not really a preference of ours, rather a meta-tool of the synthesis method).