Harley-ml/lesswrong huggingface dataset:
# AGI Safety Literature Review (Everitt, Lea & Hutter 2018)

_Abstract._ The development of Artificial General Intelligence (AGI) promises to be a major event. Along with its many potential benefits, it also raises serious safety concerns (Bostrom, 2014). The intention of this paper is to provide an easily accessible and up-to-date collection of references for the emerging field of AGI safety. A significant number of safety problems for AGI have been identified. We list these, and survey recent research on solving them. We also cover works on how best to think of AGI from the limited knowledge we have today, predictions for when AGI will first be created, and what will happen after its creation. Finally, we review the current public policy on AGI.
# Looking for AI Safety Experts to Provide High Level Guidance for RAISE

The [Road to AI Safety Excellence](https://www.lesswrong.com/posts/ATQ23FREp9S4hpiHc/raising-funds-to-establish-a-new-ai-safety-charity) (RAISE) initiative aims to allow aspiring AI safety researchers and interested students to get familiar with the research landscape effectively; thereby hopefully increasing the number of researchers that contribute to the field. To that end, we (the RAISE team) are trying to build a high-quality online course. You can see our pilot lesson [here](https://play.ihatestatistics.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL) (under “Corrigibility 1”).

Most of the course segments will be based on distilled summaries of one or more papers. We already distilled ~9 papers on corrigibility for the first course segments, and used the distilled summaries to write video script drafts.

Our long-term goal is to cover as much of the AI safety research landscape as possible, in the most useful way possible. Therefore, we need guidance from experts who have extensive familiarity with the literature in one of the broad subfields of AI safety (i.e. the machine learning perspective or the Agent Foundations research agenda; or broad parts thereof). We realize that the time of such experts is a critically scarce resource. Therefore, we will ask them only for high-level guidance including:

1) Their idea of a good structure for a part of the course: a list of sections, and the subsections that might constitute each one.

2) Pointers to papers to base each subsection on.

If an expert expects contributing further to RAISE to be an effective use of their time, they could also choose to go over our lesson scripts and provide feedback before the [videos](https://www.youtube.com/watch?v=uEtAbi6SJio) are being recorded.

Should this role be an effective use of your time, please contact us at _raise@aisafety.camp_
# Predicting Future Morality

Robin Hanson [suggests](https://www.lesswrong.com/posts/toE6i842jhkkvXy7W/predicting-future-morality#spHdwbMaEnn4qW8CA) that recent changes in moral attitudes (in the last few hundred years) are better explained by changing circumstances than by progress in moral reasoning.

This seems plausible to me. It also seems likely that there would be a bit of a lag between the change in circumstance and the common acceptance of the new morality. (The sexual revolution following the introduction of the pill seems like a good example.)

Suppose this is broadly right -- that moral attitudes follow circumstances. Is there anything we can predict about where moral attitudes will be in the next few decades (or [economic doublings](https://sideways-view.com/2018/02/24/takeoff-speeds/)), based on either recent technological or economic changes, or on those we can see on the horizon?
# Advocating for factual advocacy 

In this post I present a Hansonian view of morality, and tease out its consequences for advocacy. Beware, all of this is the result of armchair speculation.

The simplified model goes something like this:

*   Human morality is mostly a justification mechanism, trying to give coherence to the actions we were going to do anyway.
*   Concretely, when spousing a belief or its opposite is cheap (it does not fulfill a social function nor will it contradict our future actions) we will prefer to stick to the position that better fits our current plan.
*   In general our plan follows a minimal effort law, so we will stick to the side of the belief which is more convenient.
*   The other side of the coin is that when we receive new information about the world which changes what we will do in the future it has profound effects in our morality.

Example: Consider [this study](http://www.nber.org/papers/w23532) which reaches the conclusion that a higher minimum salary reduced the average payroll of a low income person by 125$. This is not a moral claim, yet for some it will feel like it is. Often we find that factual claims are processed by our brains as moral claims.

This explains why humans have so much difficulty with the "is" vs "ought" question - for the explicit models in the human brain, what is and what ought to be are both grounded on factual information.

This model makes a bold prediction: philosophical arguments such as ["The Drowning Child"](http://effective-altruism.com/ea/6u/the_drowning_child_and_the_expanding_circle/) do not derive their strength from their moral validity, but instead from the factual revelation that it is possible to save a child's life for a rather modest sum of money.

This has direct relevance to advocacy, since it suggests that in order to change somebody's moral opinion on eg animalism we should not focus on trying to make moral arguments, but instead on giving people new data they previously didn't have and that increases the perceived convenience of taking a particular action eg objective figures on how much money does it cost to save an animal's life.

You can also go one meta level up and change how convenient it factually is to help with your cause of choice, by for example developing clean, affordable meat.

This contrast starkly with people trying to use emotional appeal or philosophically grounded arguments, which we would predict to have small long term effects.

Questions: What other predictions does this model make? Where does it fail? How can it be refined? How can we apply it to specific causes such as AI Safety research?
# LW Update 5/6/2018 – Meta and Moderation

**User Facing Changes**

*   New posts can only be submitted to your personal blog. Moderators will move it to the frontpage if it seems appropriate.
*   "Personal Blogposts" tab is now labelled "All Posts", and now contains meta posts.

*   We had deliberately been pushing on "disincentivize meta conversation", and still are, but since the Meta section basically has only been used for updates from the admins, it seemed like A) we could lean a bit in the other direction, B) I think having the third tier of posts on the frontpage be "all posts" is a lot more clear about what the frontpage hierarchy is trying to do.

*   Meta posts now appear properly in your drafts on your user profile.
*   Your user profile will now only show the 4 most recent drafts

*   I rarely found I needed to see anything but my most recent drafts, and meanwhile it was a bit annoying to scroll past it whenever I went to my profile page.

**Moderator Changes**

*   Moderators now have a sidebar that shows a list of all new _unreviewed_ personal blogposts, with easy buttons for:

*   approve (and leave as a personal blogpost)
*   move to frontpage
*   delete

In the past month or so, we've a) continued seeing a rise in post volume, while b) a couple of our moderators have been more busy than usual. I've been increasingly worried about the signal/noise ratio on the site.

So this update has been focused mostly on some tools that make moderation much _easier,_ which many users won't directly experience but which I hope will lead to a better site experience.

Longterm, my goal is to put all moderator To Do items in the sidebar so they are easily seen, and then make it easy and encouraged to maintain inbox zero on the sidebar. I'd hoping this will help both reactive "notice and delete spam" type actions, and "notice and promote good things" type actions.

**Minor Naming Convention Update**

I'll be calling these posts "LW Update \[Date\] – {Description}" from now on, since "update" wasn't very each to search for, and hopefully "LW Update" will be. (We'll eventually also make a better search tool, but even when it's done it seems like it should be easy to search naively)

* * *

[Link to Git history](https://github.com/Discordius/Lesswrong2/commits/devel). Commit a7c9879b5f5fa94f0a843993b10c85898913a43e
# Introducing the Longevity Research Institute

I’ve just founded a nonprofit, the Longevity Research Institute — you can check it out [here.](https://thelri.org/)

The basic premise is: we know there are more than 50 compounds that have been reported to extend healthy lifespan in mammals, but most of these have never been tested independently, and in many cases the experimental methodology is poor.

In other words, there seems to be a _lot _of low-hanging fruit in aging.  There are many long-lived mutant strains of mice (and invertebrates), there are many candidate anti-aging drugs, but very few of these drugs have actually been tested rigorously.

  

Why?  It’s an incentives problem.  Lifespan studies for mice take 2-4 years, which don’t play well with the fast pace of publication that academics want; and the FDA doesn’t consider aging a disease, so testing lifespan isn’t on biotech companies’ critical path to getting a drug approved.  Mammalian lifespan studies are an underfunded area — which is where we come in.

  

We write grants to academic researchers and commission studies from contract research organizations.  Our first planned studies are on epitalon (a peptide derived from the pineal gland, which has been reported to extend life in mice, rats, and humans, but only in Russian studies) and C3 carboxyfullerene (yes, a modified buckyball, which prevents Parkinsonism in primate models and has been reported to extend life in mice).  I’m also working on a paper with [Vium](http://www.vium.com/) about some of their long-lived mice, and a quantitative network analysis of aging regulatory pathways that might turn up some drug targets.

  

We’re currently fundraising, so if this sounds interesting, please consider donating. The more studies that can be launched in parallel, the sooner we can get results.
# Is epistemic logic useful for agent foundations?

The title isn't a rhetorical question; I'm actually looking for answers. This summer, I'll have the opportunity to attend a summer school on logic, language and information. Whether or not I go depends to a significant extent on whether what they'll be teaching - particularly epistemic logic, also some other topics in logic and language - will be useful for AI safety research. [Here is a summary of epistemic logics](https://plato.stanford.edu/entries/logic-epistemic/), and [here are the courses I'll be able to take](https://www.cmu.edu/nasslli2018/courses/index.html). I've already taken classes in first-order logic, but right now I'm uncertain about the value of doing these extra courses.

Reasons to think learning epistemic logic will be useful for agent foundations:

*   MIRI's work relies heavily on high-level concepts in logic
*   Epistemic logic is particularly concerned with statements about knowledge and belief which seem very relevant to reasoning about agents
*   Learning about epistemic logic is probably useful for thinking about other forms of logic

Reasons to think it won't be useful:

*   As far as I can tell, it doesn't appear on MIRI's research guide, nor in any of their papers, nor in the sequences
*   It seems like epistemic logic is mostly non-probabilistic and is talking about a fundamentally different sort of knowledge from probabilistic Bayesian knowledge, which increases my credence that it's the sort of philosophy which isn't going to be of much practical use
# Upvotes, knowledge, stocks, and flows

This is a short post inspired by ["EXEMPLIFYING EQUITABLE GROWTH: MR. GOOGLE SERVES ME A BAKER'S HALF-DOZEN FROM THE WCEG WEBSITE, AND WHAT I LEARN THEREBY..."](http://www.bradford-delong.com/2018/05/exemplifying-equitable-growth-mr-google-serves-me-a-bakers-half-dozen-from-the-wceg-website-and-what-i-learn-thereby.html#more). Delong looks at google results for his blog archive, and finds they are very weakly predictive of what he thinks is useful about his blog. He thinks about that in terms of the "flow" of intellectual discussion — what people are talking about — versus the "stock" — what is known.

I think that's a productive analytical frame. Websites live and die off of their salience to the flow, but their long-term value is what they contribute to the stock. Twitter is optimized for flow to the point where it's nearly useless as stock; but high-quality work that never gets into the flow ends up so unread and hard to find that it has little value as stock.

For sites like Less Wrong, I think that thinking about stock and flow could help to improve how upvotes work. To some extent, upvotes help promote a healthy flow, which is an important and useful goal. But that is essentially a short-term value, and so upvotes like that should decay. On the other hand, upvotes also serve as long-term markers of valuable stock. That kind of upvote shouldn't decay.

One place this kind of thinking could lead would be to having different kinds of upvotes. "Thanks" and "good question" would be more flow-y and thus decay faster; "well said" and "good link" would be more stock-y and last longer.
# Tech economics pattern: "Commoditize Your Complement"

> Joel Spolsky in 2002 identified a major pattern in technology business & economics: the pattern of "commoditizing your complement", an alternative to vertical integration, where companies seek to secure a chokepoint or quasi-monopoly in products composed of many necessary & sufficient layers by dominating one layer while fostering so much competition in another layer above or below its layer that no competing monopolist can emerge, prices are driven down to marginal costs elsewhere in the stack, total price drops & increases demand, and the majority of the consumer surplus of the final product can be diverted to the quasi-monopolist.

> A classic example is the commodification of PC hardware by the Microsoft OS monopoly, to the detriment of IBM & benefit of MS.

> This pattern explains many otherwise odd or apparently self-sabotaging ventures by large tech companies into apparently irrelevant fields, such as the high rate of releasing open-source contributions by many Internet companies or the intrusion of advertising companies into smartphone manufacturing & web browser development & statistical software & fiber-optic networks & municipal WiFi & radio spectrum auctions (Google): they are pre-emptive attempts to commodify another company elsewhere in the stack, or defenses against it being done to them.
# RFC: Philosophical Conservatism in AI Alignment Research

I've been operating under the influence of an idea I call philosophical conservatism when thinking about AI alignment. I am in the process of summarizing some of the specific stances I take and why I take them because I believe others would better serve the project of alignment research by doing the same, but in the meantime I'd like to request comments on the general line of thinking to see what others think. I've formatted the outline of the general idea and reasons for it with numbers so you can easily comment on each statement independently.

1.  AI alignment is a problem with bimodal outcomes, i.e. most of the probability distribution is clustered around success and failure with very little area under the curve between these outcomes.
2.  Thus, all else equal, we would rather be extra cautious and miss some paths to success than be insufficiently cautious and hit a path to failure.
3.  One response to this is what Yudkowsky calls [security mindset](https://intelligence.org/2017/11/25/security-mindset-ordinary-paranoia/) by alluding to [Schneier's concept of the same name](https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html).
4.  Another is what I call philosophical conservatism. The ideas are related and address related concerns but in different ways.
5.  Philosophical conservatism says you should make the fewest philosophical assumptions necessary to addressing AI alignment and that each assumption should be maximally parsimonious and be the assumption that is least convenient for addressing alignment if it were true when there is nontrivial uncertainty over whether a similar, more convenient assumption holds.
6.  This is a strategy that reduces the chance of false positives in alignment research but makes the problem possibly harder, more costly, and less competitive to solve.
7.  For example, we should assume there is no discoverably correct ethics or metaethics the AI can learn since, although it would make the problem easier if this were true, there is nontrivial uncertainty around this and so the assumption which makes it less likely that alignment projects fail is to assume that ethics and metaethics are not solvable.
8.  Current alignment research programs do not seem to operate with philosophical conservatism because they either leave philosophical issues relevant to alignment unaddressed, make unclear implicit philosophical assumptions, or admit being hopeful that helpful assumptions will prove true and ease the work.
9.  The alignment project is better served by those working on it using philosophical conservatism because it reduces the risks of false positives and spending time on research directions that are more likely than others to fail if their philosophical assumptions do not hold.
# Bay Summer Solstice

\[Updated June 6th\]

Exact details are still being finalized, but here's the shape of this event. We'll be heading out to a place where we can barbecue, explore, build things together and watch the sun set on the longest day of the year.

Suggested donations are $15/$35/$50+ depending on whether you just moved to the Bay and don't have a job yet or are one of those 6-figure tech folk.

The [overall vision of the event is described here](https://www.lesswrong.com/posts/KnpxChD9fnT9785FR/visions-of-summer-solstice).

We'll be arranging a big caravan for people who want to head out together.

*   Weekend of the 16th – People who are interested in helping test-build the dome journey out for a Day Of Building The Dome, Noticing What Screws Up, And Devising Final Plans Around That
*   Friday the 22nd – People who want to held set up the day before (constructing domes, gateways and the like), and partake in some special rituals for people who do that, can spend the afternoon/evening doing that. A small number of people will camp out to watch the site. (We're currently not expecting to get the permit in time for the a larger camping trip)
*   Saturday the 23rd – A big caravan departs from Berkeley (probably arranging a few cargo vans to drive over from REACH, starting at 1pm and then departing as they fill up). Big Day of Festivities. At the end of day, cargo vans bring people back. A couple people camp out once more to watch the dome and stuff
*   Sunday the 24th – Final takedown of the setup.
# Co-Proofs

At the [recommendation of Jacobian](https://www.lesswrong.com/posts/uGdWZSeHw5uNbhu7M/book-review-too-like-the-lightning), I've been reading _Too Like the Lightening_. It is a thoughtful book which has several points of interest to rationalists (imho), but there is one concept which I think is nice enough to pluck out and discuss in itself, rather than being satisfied to suggest that people read the book. I also want to suggest a different name than the one from the book.

If you think discussion of a logical concept which is mentioned in a book is a spoiler, maybe stop here.

At one point, there is a discussion in which one character is explaining how much some other characters must already know. The term "anti-proof" is used to refer to failure to falsify a hypothesis. Having a short term for this concept seems like a really good idea. We have the phrase "absence of evidence is evidence of absence", but we don't have a word for the positive case, where absence of counter-evidence speaks in favor of a hypothesis.

Unfortunately, "anti-proof" sounds more like the former than the latter, even though it is being used for the latter in the book. A more appropriate term would be "co-proof", since it is the absence of a proof of the negation.

For example, an alibi would refute someone's involvement in a crime. The absence of an alibi, then, is a co-proof of their involvement: it does not prove involvement by any means, but it _must_ constitute some supporting evidence, by [conservation of expected evidence](https://www.lesswrong.com/posts/jiBFC7DcCrZjGmZnJ/conservation-of-expected-evidence).

By "proof of H" I mean an observation which would make the probability of H very close to 1. (How close is "very close" depends on standards of proof in a context, with mathematics demanding the highest standards.) By "refutation" I mean a proof of the negation. So, a co-proof is an observation whose negation would have taken the probability of H to very near zero:

_**E is a co-proof of H :=**_ < refresh to render LaTeX >

Why are co-proofs of interest? Popperian epistemology is the claim that scientific hypotheses can be supported only by co-proofs; we attempt to refute things, and if something has survived enough refutation attempts, it is considered to be a strong hypothesis. Bayesians are not Popperians, but Popper was still mostly right about this; so, having a short name for it seems useful.
# Mini-review: The Book of Why

Someone should probably write a real book review, but to make a brief recommendation: _The Book of Why_ by Judea Pearl and Dana Mackenzie is probably the most interesting general-science book I've read since _Thinking Fast and Slow._

Pearl's goal is to explain and promote _causal inference,_ which you might think of as (allegedly) the next big thing after frequentist and Bayesian statistics. The introduction is probably skippable, since the authors make some rather grand claims that aren't backed up until later. I found myself thinking, "okay, maybe it's great, but explain what it is already".

Chapter 1 introduces the _Ladder of Causation_, the authors' way of distinguishing the correlations found via a model-free statistical summary of data (which is level 1) from deductions that require a causal model (levels 2 and 3).

Chapters 2 and 3 give a partial, "whiggish" history of statistics from a causal perspective, covering frequentist and Bayesian statistics and Pearl's AI work, when he invented Bayesian networks. At the end, he talks about the possible junctions in a Bayesian network: the _chain_, _fork_, and _collider,_ and how they can easily cause confusion.

Chapter 4 uses causal reasoning to explain the logic behind randomized controlled trials and other ways of controlling for confounding variables.

Chapter 5 covers the scientific debate over cigarette smoking, and how lack of clarity about causation resulted in this debate taking years longer than it needed to.

Chapter 6 is a fun chapter showing how to use causal diagrams to shed new light on the Monty Hall problem and Simpson's paradox.

And that's as far as I've read, but it's enough to make a strong recommendation.

I did a quick search on Less Wrong and causality has been covered before, though not as clearly. In particular, see Yudkowsky's [Causal Diagrams and Causal Models](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models).

(I was confused about one bit, though: Yudkowsky writes that "Causal _models_ (with specific probabilities attached) are sometimes known as 'Bayesian networks' or 'Bayes nets'." But in the book, the authors make a clear distinction: "Unlike the causal diagrams we will deal with throughout the book, a Bayesian network carries no assumption that the arrow has any causal meaning." Though later, they write, "These three junctions \[...\] are like keyholes through the door that separates the first and second levels of the Ladder of Causation.")
# Ineffective entrepreneurship: post-mortem of Hippo, the happiness app that never quite was

" I spent two and half years trying to start a startup I thought might do lots of good. It failed. I explain what happened, how it went wrong and try to set out some relevant lessons for others. Main lesson: be prepared for the fact you might find the experience mostly stressful and have nothing particularly useful come of it."
# Understanding is translation

Does this feel familiar: "I thought I understood thing X, but then I learned something new and realized that I'd never really understood X?"

For example, consider a loop in some programming language:

    var i = 0;

    while (i < n) {

      i = i + 1;

    }

If you're a programmer, you probably understand it just fine. How it works, in what order the lines are executed, how the variable changes over time... But have you ever noticed that the simplest way to compile such a loop to machine code involves two jump instructions - one conditional and one unconditional? (Try doing that with only one jump, it won't work.)

Now you might feel that your "understanding" of loops has become slightly closer to "proper understanding".

Or not!

An alternative view is that understanding is translation. It's a two-place word. Your understanding of loops, in the sense of translating them to execution histories, was perfectly fine. But your understanding of loops, in the sense of translating them to machine code, was slightly lacking.

When you see that pattern once, you notice it everywhere. A middle-schooler can understand numbers, in the sense of translating them to amounts of apples and such, but doesn't immediately translate the expression "x > 5" to a half-open ray on the number line. A self-taught singer can translate from heard notes to sung notes, but can't translate either to notes on a staff; a self-taught guitarist is missing a different subset of those skills. A bilingual person can translate a Japanese sentence with the word "integral" to English, without knowing what integral means. You can be good at translating other people's facial expressions to emotional states, but lousy at translating them to pencil sketches; your friend is the opposite; which of you "understands" human faces better? There's no answer, or many answers. Don't ask whether someone understands X. Instead, ask if they can translate X <-> Y.

That has implications for teaching. If you walk into a classroom intending to make students "understand" X, you'll fail at teaching. (I'm speaking from experience here.) But if you find some Y, already understood by the students, that can be translated to X - and drill them repeatedly on both directions of translation - then they will begin to "understand" X.
# *Another* Double Crux Framework

_Epistemic Status: Untested hypothesis. Looking to operationalize better._

Followup to:

*   A [Concrete Multi Step Variant of Doublecrux I Have Used](https://www.lesswrong.com/posts/isDnrPdRaXj5Ce3dN/a-concrete-multi-step-variant-of-double-crux-i-have-used) (deluks917)
*   [Musings on Double Crux](https://www.lesswrong.com/posts/T3nb24aZf7d5S2Lnm/musings-on-double-crux-and-productive-disagreement) (me)

A problem with resolving deep disagreements is that there's a _lot_ of finnicky moving parts to the dispute. It's easy to bring up 3 examples to illustrate a point, then you get fixated on one of the examples, and then bring up some adjaecent issues to those examples, and then hours (or months) later you've managed to go in circles three times and still not really resolve anything.

I just had an idea, which may or may not work but I'd like to try (and I have something like 3-5 outstanding deep disagreements with 3-4 different people so it seemed like an opportune time to flesh it out), which is to manage the deep disagreement via a google doc, divided into sections:

1. **Common beliefs** – What do we actually agree on?

2\. **Empirical Checks / Bets** – What things you can go look at in the real world that'd resolve common cruxes?

3\. **Common cruxes** – What do we agree that we disagree on? (that we have operationalized, such that we are confident we understand each other, and can think productively about how to empirically resolve them)

4\. **Passing Each Other's [ITT](https://www.lesswrong.com/posts/PH47N6uTdtbLAHrAr/the-ideological-turing-test) –** Each person attempts to summarize each other's position as best they can, in succinct bullet point form. (Contains a subsection for each participant)

5\. **Amorphous exploratory discussion / ranting** – Two large meandering sections that attempt to suss out the parts of the disagreement where you aren't actually sure what each other are saying. Still try to keep points to individual bullets.

This can be done in tandem with an in person discussion. But the main idea here is have an _explicit goal_ that bullet points slowly get refactored into clearer versions of themselves, and move up the hierarchy. Amorphous rants should turn into ITT-passing arguments should turn into clearly identified common cruxes should turn into empiricism should turn into shared beliefs. (And, hopefully, a lot of disagreements dissolve under scrutiny such that empiricism isn't even necessary)

This would give you a clear sense of whether you're accomplishing anything – if the bottom sections keep getting bigger and don't eventually move to the upper sections, you're probably talking in circles, or making some kind of error mode. You might or might not get to the end (where _all_ you have is common beliefs), because empiricism can be expensive, but hopefully at least you transmute things up into the common crux section.
# Announcement: Legacy karma imported

Hey Everyone,

I finally got around to properly importing the old karma from LessWrong 1.0. Your vote-weight and the karma displayed on your profile should now reflect all the contributions you made to LessWrong over the years.

We might change this around again at some future point in time, but we just stuck with the same karma formula as the old LessWrong had, with old main posts being worth 10 karma per vote, and everything else being worth 1.

Sorry for this taking so long. Comment or ping us on Intercom if you notice anything karma-related being broken, or your karma not being properly imported.
# Open Thread June 2018

If it’s worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1.  Check if there is an active Open Thread before posting a new one (use search for Open Thread ).
    
2.  Monthly open threads seem to get lost and maybe we should switch to fortnightly.
    
3.  What accomplishments are you celebrating from the last month?
    
4.  What are you reading?
# Simplified Poker

This is intended as a three-part sequence. Part two will go over my strategy. Part three will reveal the results and discuss some implications.

In the same class in which we later played [The Darwin Game](https://thezvi.wordpress.com/2017/11/15/the-darwin-game/), we played a less complex game called Simplified Poker. As in The Darwin Game, we were given the rules and asked to submit instructions for a computer program that would play the game, and the professor would then code our programs for us.

The rules of Simplified Poker are as follows:

Game is played with a 3-card deck, with the cards labeled 1, 2 and 3.

Each hand, the players alternate who goes first, each player antes one chip and is dealt one card.

The first player can bet one chip, or check.

If the first player bets, the second player can either call the one chip bet, or fold.

If the first player checks, the second player can either also check, or can bet. If the second player bets, the first player can either call the one chip bet, or fold.

There is at most one bet per hand, as neither player is allowed to raise.

If either player folds, the other wins the pot of 2 chips and takes back their 1 chip bet. Neither card is shown. If neither player folds – either both players check, or there is a bet and a call – then both cards are revealed and the player with the higher card takes all 4 chips.

In the class, all programs would play a round robin with all other programs, with 50 hands per match. Your goal is to maximize the average number of chips won over all rounds – note that how many opponents you beat does not matter, only the number of chips won.

The game is simple. A lot, but far from all, of your decisions are forced. There’s no weird trick, but optimal play still isn’t obvious. I’ll pause here to allow and encourage thinking about what strategy you’d submit.
# Fundamentals of Formalisation Level 3: Set Theoretic Relations and Enumerability

Followup to [Fundamentals of Formalisation level 2: Basic Set Theory](https://www.lesswrong.com/posts/yzYSj7nzYq6pk5c6L/fundamentals-of-formalisation-level-2-basic-set-theory).

  

The big ideas:

*   Ordered Pairs
*   Relations
*   Functions
*   Enumerability
*   Diagonalization

To move to the next level you need to be able to:

*   Define functions in terms of relations, relations in terms of ordered pairs, and ordered pairs in terms of sets.
*   Define what a one-to-one (or injective) and onto (or surjective) function is. A function that is both is called a one-to-one correspondence (or bijective).
*   Prove a function is one-to-one and/or onto.
*   Explain the difference between an enumerable and a non-enumerable set.

Why this is important:

*   Establishing that a function is one-to-one and/or onto will be important in a myriad of circumstances, including proofs that two sets are of the same size, and is needed in establishing (most) isomorphisms.
*   Diagonalization is often used to prove non-enumerability of a set and also it sketches out the boundaries of what is logically possible.

  

You can find the lesson in our [ihatestatistics course](https://play.ihatestatistics.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL). Good luck!

P.S. From now on I will posting these announcements instead of [Toon Alfrink](https://www.lesswrong.com/users/toonalfrink).
# Front Row Center

Epistemic Status: Lightweight

Related: [Choices are Bad](https://thezvi.wordpress.com/2017/07/22/choices-are-bad/), [Choices Are Really Bad](https://thezvi.wordpress.com/2017/08/12/choices-are-really-bad/)

Yesterday, my wife and I went out to see Ocean’s Eight (official review: as advertised). The first place we went was a massively overpriced theater (thanks MoviePass!) with assigned seating, but they were sold out (thanks MoviePass!) so we instead went to a different overpriced theater without assigned seating, and got tickets for a later show. We had some time, so we had a nice walk and came back for the show.

When we got back, there was nowhere for us to sit together outside of the first two rows. They’re too close, up where you have to strain your neck to see the screen. My wife took the last seat we could find a few rows behind that, and I got a seat in the second row. It was fine, but I’d have much preferred to sit together.

It was, of course, our fault for showing up on time rather than early to a sold out screening. I mention it because it’s a clean example of how offering less can provide more value.

The theater should, if they don’t want to do assigned seating, rip out the first two rows.

At first this seems crazy. Many people prefer sitting in the first two rows to being unable to attend the show, so the seats create value while increasing profits. What’s the harm?

The harm is introducing risk, and creating an expensive auction.

The risk is that if you go to the movies, especially the movies you most want to see, you’ll be stuck in the first two rows. So when you buy a ticket and go upstairs, you might get a bad experience. If the show is sold out, that might be better, as you can buy a different ticket or none at all.

The auction is worse. Seats are first come, first serve. So if it’s important to get served first, you need to come first. If it’s very important to _not be last,_ to avoid awful seats, you need to come early, and so does everyone else, bidding up the price of not-last the same way you’d bid up being first.

With no awful seats, those who care a lot about better seats will still come early, but most people care a lot less. So everyone can come substantially earlier, and not feel pressure. Many will show at the last minute, and be totally fine.

The deadweight loss in time, of adding those forty extra seats, is massive, distributed throughout the theater. Everyone feels pressure to get there early even when they already have a ticket, so even if their seat is good, they stressed out about their seat, and not only burned time but feel bad about being pressured.

Avoiding time-based auctions and signals, or at least minimizing the value at stake in them, is an important and underappreciated problem.
# Announcing the second AI Safety Camp

What is the AI Safety Camp?
---------------------------

Would you like to work on AI safety or strategy research and are looking for a concrete way to get started? We are organizing this camp for aspiring AI safety and strategy researchers. At the camp, you:

*   build connections with others in the field
*   build your research portfolio
*   receive feedback on your research ideas and help others with theirs
*   make concrete progress on open AI safety research questions

Read more about the last research camp [here](https://aisafetycamp.com/previous-camps/), including a summary of the [produced research](https://aisafetycamp.com/2018/06/05/aisc-1-research-summaries/).

What’s the structure of the camp?
---------------------------------

The camp is preceded by 7 weeks of preparation in form of an online study group of 3-5 people, followed by a 10-day intensive camp with the aim of creating and publishing a research paper, extensive blog post, or github repository.

What will attendants work on?
-----------------------------

Participants will work in groups on tightly-defined research projects, for example in the following areas:

*   Strategy and Policy
*   Agent Foundations (decision theory, subsystem alignment, embedded world models, MIRI-style)
*   Value learning (IRL, approval-directed agents, wireheading, …)
*   Corrigibility / Interruptibility
*   Side Effects, Safe Exploration
*   Scalable & Informed Oversight
*   Robustness (distributional shift, adversarial examples)
*   Human Values (including philosophical and psychological approaches)

When and where?
---------------

4–14 October 2018, in Prague, Czech Republic.

Pricing
-------

Attendance is free.

Apply
-----

Applications and more information on [aisafetycamp.com](https://aisafetycamp.com/)
# A general model of safety-oriented AI development

This may be trivial or obvious for a lot of people, but it doesn't seem like anyone has bothered to write it down (or I haven't looked hard enough). It started out as a generalization of Paul Christiano's [IDA](https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616), but also covers things like safe recursive self-improvement.

    Start with a team of one or more humans (researchers, programmers, trainers, and/or overseers), with access to zero or more AIs (initially as assistants). The human/AI team in each round develops a new AI and adds it to the team, and repeats this until maturity in AI technology is achieved. Safety/alignment is ensured by having some set of safety/alignment properties on the team that is inductively maintained by the development process.
    

The reason I started thinking in this direction is that Paul's approach seemed very hard to knock down, because any time a flaw or difficulty is pointed out or someone expresses skepticism on some technique that it uses or the overall safety invariant, there's always a list of other techniques or invariants that could be substituted in for that part ([sometimes](https://www.lesswrong.com/posts/o22kP33tumooBtia3/can-corrigibility-be-learned-safely/comment/8L28cEW6bmY3EHARS) in my own brain as I tried to criticize some part of it). Eventually I realized this shouldn't be surprising because IDA is an instance of this more general model of safety-oriented AI development, so there are bound to be many points near it in the space of possible safety-oriented AI development practices. (Again, this may already be obvious to others including Paul, and in their minds IDA is perhaps already a cluster of possible development practices consisting of the most promising safety techniques and invariants, rather than a single point.)

If this model turns out not to have been written down before, perhaps it should be assigned a name, like Iterated Safety-Invariant AI-Assisted AI Development, or something pithier?
# Today a Tragedy 

Very early this morning, my friend died in a car wreck.

When my mother called to tell me, she told me an old friend had died, but then it took her 10 seconds to compose herself to tell me exactly _who._

Old friends have called and texted me throughout the day. Some seeking to console, and others to be consoled.

But it's all a tragedy.

My friends shouldn't need consolation, my mother shouldn't need to compose herself, and my friend shouldn't have died today.

A resolve arose within me:

I will solve AI alignment, and then I will work to further AGI, bringing that day closer and closer to prevent the loss of another grandson, another nephew, another friend.

But when I tried to work today, all I could do was mourn.

* * *

Will,

I'm glad I got to see you one last time in January. I'm sorry I forgot to wish you a happy birthday this year, but I've saved two dates in my calendar for you now, so I will never forget. Thanks for being my debate partner, and thanks for being my friend.

Love you. Miss you.

Logan
# Counterfactual Mugging Poker Game

Consider the following game:

Player A receives a card at random that is either High or Low. He may reveal his card if he wishes.

Player B then chooses a probability < refresh to render LaTeX > that Player A has a high card.

Player A always loses < refresh to render LaTeX > dollars. Player B loses < refresh to render LaTeX > dollars if the card is low and < refresh to render LaTeX > dollars if the card is high.

Note that Player B has been given a proper scoring rule, and so is incentivized to give his true probability (unless he makes some deal with player A).

You are playing this game as player A. You only play one time. You are looking at a low card. Player B is not trying to make a deal with you, and will report his true probability. Player B is very good at reasoning about you, but you are in a separate room, so Player B cannot read any tells unless you show the card. Do you show your card?

Since your card is low, if you show it to player B, you will lose nothing, and get the best possible output. However, if player B reasons that if you would show your card if it was low, then in the counterfactual world in which you got a high card, player B would know you had a high card because you refused to show. Thus, you would lose a full dollar in those counterfactual worlds.

If you choose to not reveal your card, player B would assign probability 1/2 and you would lose a quarter.

I like this variant of the counterfactual mugging because it takes the agency out of the predictor. In the standard counterfactual mugging, you might reject the hypothetical and think that the predictor is trying to trick you. Here, there is a sense in which you are creating the counterfactual mugging yourself by trying to be able to keep secrets.

Also, think about this example the next time you are tempted to say that someone would only [Glomarize](https://en.wikipedia.org/wiki/Glomar_response) if they had an important secret.
# Notes on a recent wave of spam

I think it is likely already clear to the mods, but just to make it common knowledge: There's a spam attack currently active, which takes the form of messages that copy-paste the beginning of another upthread comment (I assume, to look like a valid comment to someone not following closely), followed by a spam link.

This seems clear-cut enough (and unlikely to be a pattern a human would follow) that it might be possible to auto-delete with code, saving our mods the need to delete manually. I can't take this project on in the immediate future, but perhaps someone reading this can.
# Fundamentals of Formalisation Level 4: Formal Semantics Basics

Followup to [Fundamentals of Formalisation Level 3: Set Theoretic Relations and Enumerability](https://www.lesswrong.com/posts/Z92cmpcDr5aDGg68c/fundamentals-of-formalisation-level-3-set-theoretic). [First Post](https://www.lesswrong.com/posts/kK67yXhmDYwXLqXoQ/fundamentals-of-formalisation-level-1-basic-logic).

The big ideas:

*   Formal Semantics
*   Model

To move to the next level you need to be able to:

*   Evaluate the truth value of logical sentences in a given model.
*   Build models for a set of logical sentences and then use those models to deduce information about the sentences.

Why this is important:

*   An AI built on a formal system will reason based on some sort of proof and model theory. The former gives its methods of proof (which you’ll learn in the next chapter), and the latter its semantics.
*   This level will give you your first sense of what models of logical sentences look like.

* * *

You can find the lesson in our [ihatestatistics course](https://play.ihatestatistics.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL).

For every lesson you have 2 options: do the whole thing, or skip to the questions and "challenges" in the end. The latter option is for those people that suspect they already know the subject. It serves as a means of verifying or falsifying this hypothesis.

Good luck!
# Investing and Personal Finance

A few years ago, Shay Weiss set a goal for himself to become financially independent and to position himself for an early retirement (FIRE).  
  
A big part of that was learning and absorbing a lot of information:

*   Reading a lot of books about finances/investing (A Random Walk Down Wall Street, Early Retirement Extreme, The Millionaire Next Door, etc.)
*   Reading a lot blogs / listening to a lot of podcasts (Mr. Money Mustache, JL Collins, The Mad Fientist)
*   Keeping up with the discourse on the personal finance and FIRE communities of Reddit

In this presentation, Shay will present a concise and synthesized version of the learnings and insights that I've accumulated in this process, talk about motivation, show some nifty graphs, share some anecdotes and personal successes/failures, and point you in the direction of the materials that he found most useful.
# LessWrong is hiring

LessWrong is looking to hire an additional programmer. Ideally, someone with 5+ years experience who is up for a CTO role, or seems like a strong candidate to grow into such a role.

We're looking for someone excited about our longterm vision. Over the next several years, we want to build a platform for intellectual progress across the rationality, EA and x-risk communities. In addition to smoothing out the core LW user experience, we're looking to:

*   Theorize about and experiment with new features that can move the state-of-the-art of online discourse forward.
*   Help revamp other rationality and EA aligned forums (we've just started a couple projects in this vein)
*   More generally, understand and solve problems relating to communication and the intellectual-pipeline in the rationalsphere. (Especially if it looks like they could benefit from technical solutions)

We want someone with technical chops (the site is built in javascript and ReactJS, although strong experience with other languages is fine), as well as someone who can help make architectural decisions that impact the sorts of goals listed above.

Position will require you commute to Berkeley.

Ping us if you're interested (sending any of us an email is fine)

The LessWrong team is:

*   Oliver Habryka (oliver.habryka@gmail.com)
*   Raymond Arnold (raemon777@gmail.com)
*   Ben Pace (benitopace@gmail.com)

Feel free to comment here with any questions about the position or our overall vision.