Harley-ml/lesswrong huggingface dataset:
# The Feedback Problem

My Blog [Bearlamp](http://bearlamp.com.au/the-feedback-problem/): [First](http://bearlamp.com.au/the-problem-of-other-minds/), [Previous](http://bearlamp.com.au/experimental-apparatus/)

[Lesswrong](https://www.lesswrong.com/posts/PdMAZzKyb2oC8739S/the-feedback-problem): [First](https://www.lesswrong.com/posts/RowwSbcT6E3ahpXiq/the-problem-of-other-minds), [Previous](https://www.lesswrong.com/posts/twssHPn2v2gA2ieTs/the-experimental-aparatus) \-\- [Greaterwrong](https://www.greaterwrong.com/posts/PdMAZzKyb2oC8739S/the-feedback-problem): [First](https://www.greaterwrong.com/posts/RowwSbcT6E3ahpXiq/the-problem-of-other-minds), [Previous](https://www.greaterwrong.com/posts/twssHPn2v2gA2ieTs/the-experimental-aparatus)

“Let me practice my free throw from different distances so that I can throw well wherever I am in the game”. [Anders Ericsson](https://peakthebook.com/index.html) (10,000 hours guy) in *Peak,* talks about deliberate practice. He also talks about the difficulty in getting feedback.

Feedback is the hardest part of learning anything. Good feedback and you can go from chopsticks to beethoven in simple steps. Technically speaking, the “hard part” of the skill is not the part which requires you to physically press the buttons on a piano. Or the part that requires you to work out how to move the piece in tetris to where you want it to go. The part of the chess game that is moving the piece to the next location on the board. The part of poker that is reading the cards and knowing which ones you have.

Most games have a basic level of skill that isn’t that hard. Anyone can [play tennis](http://theinnergame.com/) provided they can hold a racket and swing.. Okay maybe you need eyeballs and the ability to move around a court too but the barrier isn’t much higher than that. Some skills require more, unicycle actually takes balance, that might take longer to learn, and some games are complicated like this too.

Bad feedback is also useful. From the books, [How to Measure Anything](https://www.howtomeasureanything.com/), [Superforecasters](https://goodjudgment.com/superforecasting/), and everyone in the quantified self movement… Even a shitty piece of feedback has a [Value of Information](https://en.wikipedia.org/wiki/Value_of_information) that can be valuable. One of my favourite poor pieces of feedback is when I added to my [Self Form](http://bearlamp.com.au/quantified-self-tracking-with-a-form/), “did I stick to my diet today? yes/no/maybe”. Like magic for a month I stuck to my diet and I lost 2kg. One good clean feedback measure and I made leaps. (Sure enough, eventually other problems got in my way, but it was a good start.)

The feedback problem asks, “How would I know if I am improving?”. For a musician, that might be to recording yourself playing then listen back to what you sound like. For a farmer, that might be to weigh or count the crop and compare that to last year. For a scientist that might be repeated tests for reliability, and for someone with an emotionally trauma history that might look like “I don’t feel terrible”.

Next: Emotional training model
# Costa Rica SSC Meetup

Meet at the Food Court in Multiplaza, Escazú.

_This event is part of the [SlateStarCodex Meetups Everywhere 2018](http://slatestarcodex.com/2018/07/26/meetups-everywhere-2018/). This is a great opportunity to meet other rationalists and SSC readers in person. Everyone is welcome to come!_
# Sonoma County SSC Meetup

_This event is part of the [SlateStarCodex Meetups Everywhere 2018](http://slatestarcodex.com/2018/07/26/meetups-everywhere-2018/). This is a great opportunity to meet other rationalists and SSC readers in person. The organizer has not provided any contact information, but everyone is welcome to just show up!_
# Counterfactuals, thick and thin

_Summary: There's a "thin" concept of counterfactual that's easy to formalize and a "thick" concept that's harder to formalize._

Suppose you're trying to guess the outcome of a coinflip. You guess heads, and the coin lands tails. Now you can ask how the coin _would_ have landed if you had guessed tails. The obvious answer is that it would still have landed tails. One way to think about this is that we have two variables, your guess < refresh to render LaTeX > and the coin < refresh to render LaTeX > , that are independent in some sense; so we can counterfactually vary < refresh to render LaTeX > while keeping < refresh to render LaTeX > constant.

But consider the variable < refresh to render LaTeX > XOR < refresh to render LaTeX > . If we change < refresh to render LaTeX > to tails and keep < refresh to render LaTeX > the same, we conclude that if we had guessed tails, the coin would have landed heads!

Now this is clearly silly. In real life, we have a causal model of the world that tells us that the first counterfactual is correct. But we don't have anything like that for logical uncertainty; the best we have is logical induction, which just give us a joint distribution. Given a joint distribution over < refresh to render LaTeX > , there's no reason to prefer holding < refresh to render LaTeX > constant rather than holding < refresh to render LaTeX > XOR < refresh to render LaTeX > constant. I want a thin concept of counterfactuals that includes both choices. Here are a few definitions, in increasing generality:

1\. Given independent discrete random variables < refresh to render LaTeX > and < refresh to render LaTeX > , such that < refresh to render LaTeX > is uniform, a _thin counterfactual_ is a choice of permutation < refresh to render LaTeX > of < refresh to render LaTeX > for every < refresh to render LaTeX > .

2\. Given a joint distribution over < refresh to render LaTeX > and < refresh to render LaTeX > , a _thin counterfactual_ is a random variable < refresh to render LaTeX > independent of < refresh to render LaTeX > and an isomorphism of probability spaces < refresh to render LaTeX > that commutes with the projection to < refresh to render LaTeX > .

3\. Given a probability space < refresh to render LaTeX > and a probability kernel < refresh to render LaTeX > , a _thin counterfactual_ is a probability space < refresh to render LaTeX > and a kernel < refresh to render LaTeX > such that < refresh to render LaTeX > .

There are often multiple choices of thin counterfactual. When we say that one of the thin counterfactuals is more natural or better than the others, we are using a _thick_ concept of counterfactuals. Pearl's concept of counterfactuals is a thick one. No one has yet formalized a thick concept of counterfactuals in the setting of logical uncertainty.
# Sandboxing by Physical Simulation?

This is a simple idea. I do not remember if I've seen it anywhere. It is probably not original, but I am mildly surprised that I haven't come across it, even if only to see it refuted. If this is an old/dumb idea, please let me know and I'll delete it.

People have built universal Turing machines in Minecraft. It is straightforward to build a virtualized Turing machine by simulating a physical system which carries out the mechanical actions needed to instantiate one. You could obviously build a computer in a simulated physics simpler than Minecraft, but the Minecraft example is a more vivid one.

I don't even want to guess how much more computationally expensive it would be to run an AI, and the AI's simulated environment, on a Turing machine being run on a simulated physical computer, itself being simulated on mundane hardware. But it does strike me that an AI should have significantly more trouble hacking its way out of this kind of sandboxing.

It would likely have more trouble noticing that it's in a simulation in the first place, and if it did notice, it would likely have a much harder time doing anything about it. Our hardware relies on physical laws that we don't full understand. It makes errors that we sometimes can't really notice or stop, because fundamentally it's made of atoms and fields. One can imagine ways in which a clever being might intentionally strain physical hardware to see how it reacts. We already know that an attacker can induce a bit flip in neighboring memory through physics trickery. In contrast, the underlying simulated physics of the AI world/brain could be extremely simple and designed to be as free of surprises as possible. Within the doubly-simulated world, and within its own simulated brain, there would be no stray EM fields, no Internet ports accidentally left open. The AI could bang on the strata of its physics all it wanted, and all "we" would see would be the flickering light of the [Redstone](https://minecraft.gamepedia.com/Redstone) machine.

I'm not one to underestimate a superintelligence, but the baseline security of this kind of double-Sandboxing feels qualitatively different than that of physical hardware.
# Open Thread August 2018

If it’s worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1.  Check if there is an active Open Thread before posting a new one (use search for Open Thread ).
2.  What accomplishments are you celebrating from the last month?
3.  What are you reading?
4.  What reflections do you have for yourself or others from the last month?
# What are your plans for the evening of the apocalypse?

If everyone found out for sure that the world would end in five years, what would happen?

My guess is that it would take time before anything big happened. Finding out about the end of the world, that’s the kind of a thing that you need to digest for a while. For the first couple of days, people might go “huh”, and then carry on with their old routines while thinking about it.

A few months later, maybe there still wouldn’t be all that much change. Sure, people would adjust their life plans, start thinking more near-term, some would decide not to go to college after all. But a lot of people _already_ don’t plan much beyond a couple of years; five years is a long time, and you’ll still need to pay your bills until the Apocalypse hits. So many people might just carry on with their jobs as normal; if they were already doing college, well, you need to pass the time until the end of the world _somehow_. Might as well keep studying.

Of course, some people would have bigger reactions, right from day one. Quit their unsatisfying job, that kind of thing. People with a lot of savings might choose this moment to start living off them. And as the end of the world got closer and closer, people might get an increasingly relaxed attitude to work; though there might also be a feeling of, we’re all in this together, let’s make our existing institutions work until the end. I could imagine doctors and nurses in a hospital, who had decided that they want to make sure the hospital runs for as long as it can, and to make sure that nobody has to die before they really have to.

But I could also imagine, say, the waiter at some restaurant carrying on, serving customers even on the night of the apocalypse. (Be sure to make a reservation, we expect to have no free tables that evening.) Maybe out of principles, maybe out of professional pride, but maybe just out of habit.

I’m guessing there would be gradual changes to society, with occasional tipping points when a lot of people decided to stop whatever they had been doing and that created a chain reaction of others doing so as well. But it seems really hard to guess for how long things would remain mostly normal.
# LW Update 2018-08-03 – Comment Sorting

**Comment Sorting**

*   Comments can now be sorted by "oldest"
*   Old posts from the time-before-nested comments now default to sorting comments by "oldest."
*   You can set your profile preferences for default-comment sorting. Comments will attempt to sort by the following algorithm:

*   If you've manually set the comment ordering, it'll use that first.
*   If the page has a default sorting (see previous bullet), it defaults to that
*   If you've set a default sorting preference for yourself, it'll use that
*   Otherwise defaults to sorting comments by highest karma first.

**Comment Tooltips**

*   Added tooltips back to comment voting (mousing over upvote/downvote clarifies how to strong vote, and mousing over the karma total tells you the number of votes).

*   We had removed this because the old tooltip component took a long time to render (a problem on posts with hundreds of comments), but it should be fine now.

**AlignmentForum**

*   On the comment sorting options, you can now choose "magical sorting (include LW comments)" which lets you see LessWrong comments as well as Alignment Forum ones.

*   (On AlignmentForum, you can't vote on LW comments)

*   Creating a post or comment now properly auto-upvotes by default, as intended.
*   All AlignmentForum users can move comments from on alignment-posts over to the AlignmentForum version of that post. (That comment can then be upvoted and start accruing alignment karma for it's author. The commenter _can't_ post comments of their own unless an admin has made them an AlignmentForum user)

**Bug Fixes**

*   Facebook links should now properly show the post title and metadata, as intended. (Accidentally broke this in the last update)
*   Event pages now list contact info if that's been included.
# A Short Note on UDT

  

In my [last post](https://www.lesswrong.com/posts/AKkFh3zKGzcYBiPo7/logical-counterfactuals-for-perfect-predictors), I stumbled across some ideas which I thought were original, but which were already contained in [UDT](https://wiki.lesswrong.com/wiki/Updateless_decision_theory). I suspect that was because these ideas haven’t been given much emphasis in any of the articles I’ve read about UDT, so I wanted to highlight them here.

We begin with some definitions. Some inputs in an Input-Output map will be possible for some agents to experience, but not for others. We will describe such inputs and the situations they represent as conditionally consistent. Given a particular agent, we will call a input/situation compatible if the agent is consistent with the corresponding situation and incompatible otherwise. We will call agents consistent with a conditionally consistent input/situation compatible and those who aren’t incompatible.

We note the following points:

*   UDT uses an Input-Output map instead of a Situation-Output map. It is easy to miss how important this choice is. Suppose we have an input representing a situation that is conditionally consistent. Trying to ask what an incompatible agent does in such a situation is problematic or at least difficult as the [Principle of Explosion](https://www.wikiwand.com/en/Principle_of_explosion) means that all such situations are equivalent. On the other hand, it is much easier to ask how the agent responds to a sequences of inputs representing an incompatible situation. The agent must respond somehow to such an input, even if it is by doing nothing or crashing. Situations are also modelled (via the Mathematical Intuition Function), but the point is that UDT models inputs and situations separately.
*   Given the previous point, it is convenient to define an agent’s counterfactual action in an incompatible situation as its response to the input representing the situation. For all compatible situations, this produces the same action as if we’d simply asked what the agent would do in such a situation. For conditionally consistent situations the agent is incompatible with, it explains the incompatibility: any agent that would respond a certain way to particular inputs won’t be put in such a situation. (There might be conditionally consistent situations where compatibility isn’t dependent on responses to inputs, ie. only agents running particular source code are placed in a particular position, but UDT isn’t designed to solve these problem)
*   Similarly, UDT predictors don’t actually predict what an agent does in a situation, but what an agent does when given an input representing a situation. This is a broader concept that allows them to predict behaviours in inconsistent situations. For a more formal explanation of these ideas, see [Logical Counterfactuals for Perfect Predictors](https://www.lesswrong.com/posts/AKkFh3zKGzcYBiPo7/logical-counterfactuals-for-perfect-predictors).
# SSC Meetups Everywhere 2018

Scott over at SlateStarCodex is running another round of "[SSC Meetups Everywhere](http://slatestarcodex.com/2018/08/09/ssc-meetups-2018-times-and-places/)" and since currently a record-high of 80+ meetups are registered for the coming month, I figured we should help people find their nearest meetup, and make them aware of the opportunity. As part of that, we are adding the meetup map to the frontpage for the next week.

If you ever wanted to meet other rationalists/EAs/SSC-readers in person but found that there were no nearby regular meetups, or want to kickstart your local community, this is probably the best opportunity you are going to get this year.
# LW Update 2018-08-10 – Frontpage map, Markdown in LaTeX, restored posts and reversed spam votes

Here are the updates I pushed in the last week:

*   The community map is on the frontpage for the next week, to celebrate and promote the "SSC Meetups Everywhere"
*   We are now using a new Markdown processor, which allows you to use Markdown in your LaTeX. We use Pandoc's syntax, which uses single `$` , but doesn't parse them when there is a number right after the second one. I.e. `$4000 and $4000` doesn't get parsed as LaTeX since the second dollar sign has a number immediately after it.
*   I've finally figured out how the old LW database stored some of the old deleted posts, which means you can now access a bunch of posts from LW again. Most notably [this one](https://www.lesswrong.com/posts/EFQ3F6kmt4WHXRqik/ugh-fields), which many people requested.
*   I've also went back through the old database and did a larger hunt for spam and troll voters, and reversed all of their votes. This resulted in the votes of over 500 accounts getting reversed, with over 15000 votes being reversed (which is a lot, about as many votes as we had in all of March this year). If you had any of your old posts or comments downvoted, now might be the time to check whether that was just a result of trolls.
# Is there a practitioner's guide for rationality?

Hi everyone, I'm new to the community, and am currently working my way through the sequences — yes, all of them.

In the introduction to the first book of Rationality A-Z, Eliezer says:

> I didn’t realize that the big problem in learning this valuable way of thinking was figuring out how to practice it, not knowing the theory. I didn’t realize that part was the priority; and regarding this I can only say “Oops” and “Duh.” Yes, sometimes those big issues really are big and really are important; but that doesn’t change the basic truth that to master skills you need to practice them and it’s harder to practice on things that are further away. (Today the Center for Applied Rationality is working on repairing this huge mistake of mine in a more systematic fashion.)

Just wanted to ask if CFAR has got any of those reorganised materials up, and if they're linked to from anywhere on this site? Any links to _other_ rationality-as-practice blog posts or books or sequences would also be incredibly appreciated!
# Request for input on multiverse-wide superrationality (MSR)

> I am currently working on a research project as part of CEA’s summer research fellowship. I am building a simple model of so-called “multiverse-wide cooperation via superrationality” (MSR). The model should incorporate the most relevant uncertainties for determining possible gains from trade. To be able to make this model maximally useful, I would like to ask others for their opinions on the idea of MSR. For instance, what are the main reasons you think MSR might be irrelevant or might not work as it is supposed to work? Which questions are unanswered and need to be addressed before being able to assess the merit of the idea? I would be happy about any input in the comments to this post or via mail to johannes@foundational-research.org.
# [Feature Idea] Epistemic Status

_Epistemic Status: My best guess_ _(but, [epistemic effort](https://www.lesswrong.com/posts/oDy27zfRf8uAbJR6M/epistemic-effort) was "talked to like 2-3 people about it and it felt good to each of us")_

I've been thinking about a number of issues relating to "what are the standards supposed to be on LessWrong, anyway?", with an understanding that there are different stages of idea development. I'd like the site to have a (valid) reputation for being epistemically sound. I also think that often in the early stages of idea-formation, it's important to be in a looser, playful brainstormy mindset.

A possibility is to actually make Epistemic Status a formal field in the Post submission form. In theory people can just do this fine without special fields, but having it be an official part of the site helps steer the culture of the site.

I'm currently imagining two fields: an freeform text field that people can write whatever they want into, and a multiple-choice field with a few specific options, most likely being:

*   **Empty** (both for backwards compatibility and for posts that for whatever reason don't neatly fit into the epistemic status framework)
*   **Exploratory** (for brainstorming, off-the-cuff or other early stage idea generation. Correspondingly, critiques of these posts should lend themselves to a more playful/brainstormy atmosphere)
*   **My Best Guess** (for when you're pretty sure the idea is actually good and are now ready for serious critique)
*   **Authoritative** (for when you are making a strong or clear-cut claim that you are confident is well backed by legible evidence)

The main cost of this is that every additional UI element comes with some complexity, and small bits of complexity can add up over time. But I think this would be good both for shaping how people approach writing on LessWrong, as well as commenting.
# Large Research Report on Attitudes to Genetic Technologies

Dear All,

The Royal Society has commissioned a large 'dialogue' research report on people's attitudes to Genetic Technologies. This is significant, as CRISPR-Cas9 is a potential high-impact intervention: it's cheap, and can produce heredetary changes.

To my knowledge this is one of the deepest pieces of research ever conducted on this issue, and I think it might merit reading large sections of it, for those that are interested in the area.

For those just somewhat interested I recommend downloading the Appendices, and looking, in particular, at Appendix 8, which contains quantitative data. With n=2,061, this is a large survey.

You might jump directly to Q12, on p. 64, where respondents are asked the extent to which various developments would be positive or negative for society, or maybe to Q13, on p. 65, where they are asked "To what extent do you agree that genetic technologies such as genome sequencing and editing should be used in humans \*\*for prolonging life\*\* beyond current life expectancies" (emphasis mine).

54% agree either strongly or to some extent; that was surprising to me.

Q15, p. 66, ask about \*enhancing abilities\*, e.g. intelligence, and there \*\*only 32% agree\*\* strongly or to some extent; \*\*60%\*\* disagree.

And there's many other interesting questions.

A 2015 survey from France is here: [https://www.ifop.com/publication/les-francais-et-la-technique-du-crispr-cas9/](https://www.ifop.com/publication/les-francais-et-la-technique-du-crispr-cas9/)
# Isolating Content can Create Affordances

Recently, I've noticed an interesting failure mode in some communities I've observed online.

In this scenario, there would be some kind of content that appealed to certain users, but that many others wanted to avoid seeing and indeed considered undesirable (gossip/drama, angry political arguments, sexually explicit content, etc.). In order to prevent this content from "bleeding into" conversation, administrators would create new channels where this sort of thing could be quarantined away from normal discussion (or repurpose existing channels to sequester this content), thus allowing people who wanted to avoid this sort of content to avoid having to deal with it.

However, at least from my perspective this sometimes had a bit of an ironic effect. Once the "containment area" was established, there was now an obvious [affordance](https://en.wikipedia.org/wiki/Affordance) for this type of content that did not exist previously. As a result, the quarantined content would actually become much more prevalent in the community - further, in some cases newcomers would join the community looking for one thing, see the quarantine channel, and think something along the lines of "wow, I never knew this was a community for <insert undesirable content here>!"

The net effect of this was that the quarantines often backfired; meant to isolate undesirable or controversial content, they instead made that content more prevalent in the community as a whole, even if people no longer had to encounter it in general channels.

The chief lesson I draw from this is that it can be better to simply forbid unwanted content rather than creating a quarantine for it. If there are truly people coming to your community in large part for something that many would rather not have, you may not want them there; further, there may be those who are driven away by the presence of such content, and insofar as the undesirable content presence increases the latter group may become significantly larger.
# LW Update 2018-08-23 – Performance Improvements

Today's deploy includes:

*   Removed the map of meetups from frontpage since the major Slatestar Everywhere push is wrapping up.
*   Posts with lots of comments should load faster now
*   Fixed the bug where the last post in a sequence wouldn't properly navigate to the next post in some cases.
*   Printing a page now should work more nicely without stylesheet glitches.
*   Continued refactoring our style management in preparation for EA Forum and AI Alignment Forum theming. In the process, we:

*   streamlined some of our font sizes
*   updated our Form styling
*   made most UI elements sans-serif

In some cases the styling refactor has caused some edge cases and less common sections of the site to not quite render properly, which we apologize for and are continuing to work on.
# I am Sailor Vulcan!

Greetings! I am Sailor Vulcan, champion of justice and reason! In the name of the Moon--uh, I mean...

Hi, I'm Harry. I'm a reader, writer and gamer with a passion for rationality and existential risk prevention.

I sometimes jokingly compare my life to an intelligence explosion. I honestly believe that if there was a contest for most improved person in the world, I'd be in the running for it. When I was a kid I was a screaming, incoherent moron and lunatic. When I got to college I was still a screaming, incoherent moron and lunatic, but significantly less so. The older I've gotten, the more quickly I've improved. Most people, even other rationalists, would probably be shocked by how much I've improved myself within the past couple years, let alone the past four or five years or even the last ten.

I read HPMOR and started reading the sequences towards the end of my sophomore year in college in Spring 2015, and very shortly after that I helped solve a criminal case that had stumped a licensed private investigator, despite having zero experience with detection.

I love reading and writing, and I especially love rational fiction. I'm currently writing an original rational/scifi novelette series called "Earthlings: People of the Dawn", to help promote rationality education, effective altruism and existential risk prevention.

I also have written a bunch of stuff on my blog too, including [a rationalist children's story](http://sailorvulcansstarship.blogspot.com/2017/10/a-tale-of-four-moralities.html), [a second-person rationalist/science fiction monologue](http://sailorvulcansstarship.blogspot.com/2017/02/a-strange-computer.html), [another second-person rationalist monologue](http://sailorvulcansstarship.blogspot.com/2017/06/only-sane-man.html), and some rationalist poems among other things.

I've also written quite a few rationalist/transhumanist songs. But since the only instrument I know how to play is my own voice and I don't know standard music notation I end up having to memorize all the tunes and lyrics I write and sing them a Capella style, unfortunately. I've heard a lot of people say things along the lines of that they don't think it's possible to write good rationalist music because they expect it to be super wordy and complicated and cringey, but most of my rationalist/transhumanist songs aren't like that.

Also, I have a communication learning disability, so if I ever say or do anything to make you feel upset or uncomfortable feel free to let me know (although you don't have to, but it would be super helpful).

I used to be on here under a different username, but that was a long time ago and I barely posted anything and what I did post was not very good quality, and I wanted a different less terrible username. So here I am.

Pleased to meet you all!
# Changing main content font to Valkyrie? 

(Probably not interesting to 90% of people, but would love to get input from our local typography nerds)

We've been using [Warnock Pro](https://typekit.com/fonts/warnock-pro) as our default font for body-text and content-related elements for a while, and I've been quite happy with it on my Mac, but on Linux on Windows machines the font renders as a bit spindly and thin, and can be hard to read at some smaller font-sizes.

Given that we've been refactoring a bunch of our Typography anyways, it seems like the best time to maybe switch to a new font. In most typography-matters I tend to defer to [Butterick's Practical Typography](https://practicaltypography.com/) who has a [good set of recommendations](https://practicaltypography.com/palatino-alternatives.html) for fonts similar to Warnock, and my favorite one I've found so far is Butterick's one creation [Valkyrie](https://practicaltypography.com/valkyrie.html). So changing to that seems like a thing we might do in the next few weeks.

Stylistically, my favorite font continues to be [ETBook](https://github.com/edwardtufte/et-book), but that one sadly has a [variety of](https://github.com/edwardtufte/et-book/issues/20) [rendering issues](https://github.com/edwardtufte/et-book/issues/21) that make it impossible to use on Lesswrong. Valkyrie is about as good as Warnock in that respect, but renders much better on Windows and Linux machines in that respect, but I would love to hear recommendations for other fonts if people have any good ones (commercial fonts are fine, as long as they don't come with weird "amount of user" limits).
# REACH Panel Nomination Form

The [Berkeley REACH](http://berkeleyreach.org/) is seeking [nominations](https://goo.gl/forms/Xi0wqLOY0ixecEi43) for members of a panel to help resolve interpersonal issues that arise in or around REACH or the communities it was created to serve. See the [REACH Panel Mandate](https://docs.google.com/document/d/17tifNmxiaZQJdfHjpyuPO2s2PIifHY5R4S1NzXZ6uYA/edit#heading=h.6n4v0ocs4or5) document for more details about the panel.

Please submit one [nomination](https://docs.google.com/forms/d/e/1FAIpQLSfR4t-ytY7dMn5I1Sn6pBa5A4pmtQsCRzdhEwsqmd8E9F0CIw/viewform) per qualified candidate!

Nominations are open until September 7th, 2018.
# Corrigibility doesn't always have a good action to take

In a previous critique of [corrigibility](https://intelligence.org/files/Corrigibility.pdf), I brought up the example of a corrigible AI-butler that was in a situation where it was [forced to determine the human's values](https://www.lesswrong.com/posts/T5ZyNq3fzN59aQG5y/the-limits-of-corrigibility) through its actions - it had no other option.

Eliezer pointed out that, in his view of corrigibility, there could be situations where the AI had no corrigible actions it could take - where, in effect, all it could do was say "I cannot act in corrigible way here".

This makes corrigibility immune to my criticism in the previous post, while potentially opening the concept up to other criticisms - it's hard to see how a powerful agent, whose actions affect the future in many ways, including inevitably manipulating the human, can remain corrigible AND still do something. But that's a point for a more thorough anaylsis.
# [Paper] The Global Catastrophic Risks of the Possibility of Finding Alien AI During SETI

\[edit: it looks like immediately after publishing the paper, the journal becomes extinct, so the link is no longer working\]

My article on the topic has been finally [published](http://www.jbis.org.uk/paper.php?p=2018.71.71), 10 years after first draft. I have discussed the problem [before](https://www.lesswrong.com/posts/Jng2cZQtyuXDPihNg/risks-of-downloading-alien-ai-via-seti-search) on LW. The preprint, free of paywall, is [here](https://philpapers.org/rec/TURTRC).

The main difference between the current version and my previous post is that I concluded that such attack is less probable, because if we take into account distribution in the Universe of the naive-our-level-civilizations and civilizations which has powerful AI and are SETI-senders, when the attack become possible, only if most naive civilizations go extinct before the creation of the AI. In that case, succumbing to a SETI-attack may be net positive, as the chance that it is a message from a benevolent alien AI becomes our only way to escape inevitable extinction. Anyway, we should be cautious, if we get any alien message, especially if it will have descriptions of computers and programs to them.
# 2 Events with Christine Peterson in Vancouver This Week!

\[Note: if you are not or will not be in/around Vancouver, Canada on Friday or Saturday this weekend, ignore this post.\]

\[Meta: I would usually reserve announcements like this to Facebook events for the relevant groups, or an event on the LessWrong 'Community' page. However, Christine Peterson, co-founder of the Foresight Institute informed me on short notice on Monday she is in town for 4 days this week. To maximize visibility for these rare opportunities on such short notice, I'm posting the invitation here as well.\]

Christine Peterson is a co-founder of the Foresight Institute and a host of the Personalized Life Extension Conferences, with a wide background ranging from nanotechnology to futurism.

On Friday night, she will be there for a general meet and greet, and a discussion over dinner on 2 topics: 'Effective Altruism and Long-Term Goals,' and 'Finding Your EA Life Partner.' The event starts at 5 p.m., at 6939 Frederick Ave. in Burnaby, near Metrotown. ([LessWrong event](https://www.lesswrong.com/events/hgPkXadGEu7ZPgLwK/discussion-with-christine-peterson-of-the-foresight)) ([Facebook event](https://www.facebook.com/events/679529869087527/))

On Saturday morning, Christine will be joining the BC Life Extension Hikes group, hosted by the Lifespan Society of BC We'll be meeting at the bus stop at West Georgia and Denman Street downtown. From the [Facebook event](https://www.facebook.com/events/1816574085087414/):

> Join us for a 9 km walk around the Stanley Park Seawall with Christine Peterson. She has hosted the Personalized Life Extension Conferences and has a broad background ranging from nanotechnology to futurism. We'll meet at the bus stop and depart promptly at 10am.

For the Life Extension Hike, please RSVP at either the Facebook or [Meetup.com](https://www.meetup.com/LifespanBC/events/254193751/) event.

We hope to see you there!
# Computational complexity of RL with traps

_I briefly describe an open (to the best of my knowledge) problem and link to cstheory.stackexchange questions with more details. The readers are invited to try solving those questions._

In the comment section of my [research agenda](https://agentfoundations.org/item?id=1816) essay, Jessica and I [discussed](https://agentfoundations.org/item?id=1838), among other things, whether intelligence should be defined by some sort of diminishing regret condition or just by Bayes-optimality. In particular, the former approach runs into a problem when traps are allowed, whereas the latter is still perfectly well-defined.

However, one problem with Bayes-optimality is its computational complexity. Without traps, Bayes-optimality is feasible in the weak sense that there is asymptotically Bayes-optimal algorithm (e.g. PSRL: Posterior Sampling Reinforcement Learning) that runs in time polynomial in the size of the problem, if the prior is described by an explicit list of MDP transition kernels and reward functions (so, in practice it's only feasible for a small number of small MDPs; there are also algorithms that can work for a large number of small MDPs). Hopefully this weak feasibility can be boosted into something stronger using additional assumptions about the prior/environment (e.g. hierarchical structure; presumably, deep learning somehow exploits such assumptions). However, when traps are allowed, it's not clear that even weak feasibility is achievable. In particular, PSRL is no longer asymptotically Bayes-optimal, since it is more or less guaranteed to walk into any trap given enough time.

I condensed the problem into two succinct questions on cstheory.stackexchange (but received no answers so far). In the stochastic case, there are some computational hardness results in the literature, but they are not sufficiently strong to rule out an asymptotically Bayes-optimal algorithm (they only rule out an _exactly_ Bayes-optimal algorithm). In the [deterministic case](https://cstheory.stackexchange.com/questions/41396/is-bayes-optimal-rl-of-a-finite-set-of-dfas-feasible), there are no hardness results that I know of. So, currently I would not be very surprised neither by a positive nor a negative answer to any of those questions.

EDIT: [This](https://cstheory.stackexchange.com/a/41561/7030) answer resolves the question, more or less. Even in the deterministic case, the problem is NP-hard, with a bound of at least $\Omega\left(\frac{1}{\#\{\text{hypotheses}\}}\right)$ on the approximation accuracy.
# Do what we mean vs. do what we say

_Written quickly after a CHAI meeting on the topic, haven't thought through it in depth._

If we write down an explicit utility function and have an AI optimize that, we expect that a superintelligent AI would end up doing something catastrophic, not because it misunderstands what humans want, but because it doesn't care -- it is trying to optimize the function that was written down. It is doing what we said instead of doing what we meant.

An approach like [Inverse Reward Design](https://arxiv.org/abs/1711.02827) instead says that we should take the human's written down utility function as an observation about the true reward function, and infer a distribution over true reward functions. This agent is "doing what we mean" instead of doing what we said.

This suggests a potential definition -- in a "do what we mean" system, the thing that is being optimized is a latent variable, whereas in a "do what we say" system, it is explicitly specified. Note that "latent" need not mean that you have a probability distribution over it, it just needs to be hidden information. For example, if I had to categorize iterated distillation and amplification, it would be as a "do what we mean" system where the thing being optimized is implicit in the policy of the human and is never made fully certain.

However, this doesn't imply that we want to build a system that exclusively does what we mean. For example, with IRD, if the true reward function is not in the space of reward functions that we consider (perhaps because it depends on a feature that we didn't have), you can get arbitrarily bad outcomes (see the [problem of fully updated deference](https://arbital.com/p/updated_deference/)). One idea would be to have a "do what we mean" core, which we expect will usually do good things, but have a "do what we say" subsystem that adds an extra layer of safety. For example, even if the "do what we mean" part is completely sure about the human utility function and knows we are making a mistake, the AI will still shut down if we ask it to because of the "do what we say" part. This seems to be the idea in MIRI's version of [corrigibility](https://intelligence.org/files/Corrigibility.pdf).

I'd be interested to see disagreements with the definition of "do what we mean" as optimizing a latent variable. I'd also be interested to hear how "corrigibility" and "alignment" relate to these concepts, if at all. For example, it seems like MIRI's [corrigibility](https://intelligence.org/files/Corrigibility.pdf) is closer to "do what we say" while Paul's [corrigibility](https://ai-alignment.com/corrigibility-3039e668638) is closer to "do what we mean".
# Psychology Replication Quiz

The great guys over at 80,000 Hours made a [quiz](https://80000hours.org/psychology-replication-quiz/) for that recent replication study (previously [discussed on LW](https://www.lesswrong.com/posts/7w2tCRocphK9yW8rG/unknown-knowns)), where you can find out if you can predict as well as the scientists or the prediction market. Description:

> Can you guess which psychology experiments had correct findings and which were bogus, just from reading a brief description of their results?

> Depending on how long you want to play, we'll describe 10, 15 or 21 psychology findings published in _Nature_ and _Science_, and you'll have to guess whether a replication, with a much larger sample size, got the same result.

> Before starting, the people who organised these 21 replications asked expert psychologists to predict which results would hold up. We'll show you how you compare to their performance at the end! (And give you links to all the papers.)
# Open Thread September 2018

If it’s worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

2. What accomplishments are you celebrating from the last month?
3. What are you reading?
4. What reflections do you have for yourself or others from the last month?
5. What have you tried out this month?
6. (Teaser for my next post) What is your relationship with yourself?
# Moral realism and AI alignment

“**Abstract**”: Some have claimed that moral realism – roughly, the claim that moral claims can be true or false – would, if true, have implications for AI alignment research, such that moral realists might approach AI alignment differently than moral anti-realists. In this post, I briefly discuss different versions of moral realism based on what they imply about AI. I then go on to argue that pursuing moral-realism-inspired AI alignment would bypass philosophical and help resolve non-philosophical disagreements related to moral realism. Hence, even from a non-realist perspective, it is desirable that moral realists (and others who understand the relevant realist perspectives well enough) pursue moral-realism-inspired AI alignment research.
# nostalgebraist - bayes: a kinda-sorta masterpost

Extended criticism of "Bayesianism" as discussed on LW. An excerpt from the first section of the post:

> Like most terms ending in -ism, \[Bayesianism\] can mean a number of different things.  In its most limited sense, “Bayesianism” is a collection of technical/mathematical _machinery_, analogous to a tool or toolbox.  This collection, which I will call “the Bayesian machinery,” uses a particular way of representing knowledge, and if you can represent your knowledge in that way, the machinery tells you how to alter it when presented with new evidence.

> The Bayesian machinery is frequently used in statistics and machine learning, and some people in these fields believe it is very frequently the right tool for the job.  I’ll call this position “weak Bayesianism.”  There is a more extreme and more philosophical position, which I’ll call “strong Bayesianism,” that says that the Bayesian machinery is the _single correct way_ to do not only statistics, but science and inductive inference in general – that it’s the “aspirin in willow bark” that makes science, and perhaps all speculative thought, work insofar as it _does_ work.  (I.e., if you’re doing these things right, you’re being Bayesian, whether you realize it or not.)

> Strong Bayesianism is what E. T. Jaynes and Eliezer Yudkowsky mean when they say they are Bayesians.  It is usually what I am talking about when I say I “don’t like” Bayesianism.  I think strong Bayesianism is dead wrong.  I think weak Bayesianism may well be true, in that the Bayesian machinery may well be a very powerful set of tools – but I want to understand why, in a way that defines the power of a tool by some metric other than how Bayesian it is.

Contents of the post:

**0\. What is “Bayesianism”?**

**1\. What is the Bayesian machinery?**

**1a.** **Synchronic Bayesian machinery**

**1b.** **Diachronic Bayesian machinery**

**3\.  What is Bayes’ Theorem?**

**4\. How does the Bayesian machinery relate to the more classical approach to statistics?**

**5\. Why is the Bayesian machinery supposed to be so great?**

**6\. Get to the goddamn point already.  What’s** **_wrong_ with Bayesianism?**

**7\. The problem of ignored hypotheses with known relations**

**7b. Okay, but why is this a problem?**

**8\. The problem of new ideas**

**8b. The natural selection analogy**

**9\. Where do priors come from?**

**10\. It’s just regularization, dude**

**11\. Bayesian “Occam factors”**
# AI Governance: A Research Agenda

Allan Dafoe, head of the AI Governance team at FHI has written up a research agenda. It's a very large document, and I haven't gotten around to reading it all, but one of the things I would be most excited about is people in the comments quoting the most important snippets that give a big picture overview over the content of the agenda.
# I am the very model of a self-recursive modeler

I am the very model of a self-recursive modeler

My consciousness encompasses itself in many meta layers

To Russel’s paradox, I have the set of all the answers

I cut the hair of all the non-self-barbering hairdressers

I think about me thinking of my thoughts with regularity

In my Cartesian theater, I see myself with clarity

I tell you that I lie without the slightest contradiction

The story of my life is told in meta-metafiction

My inner simulator replicates the total universe

I simulate myself as I am writing out this line of verse

My secrets are unknown to me, I keep them confidential

Referring to myself I’d say I’m quite self-referential

My strange loop’s Escherer than yours, it’s Bacher and it’s Godeler

I am the very model of a self-recursive modeler

* * *

Cross-posted in full from [Putanumonit](https://putanumonit.com/2018/08/28/very-model/)
# Archiving link posts?

**[Link rot is a huge problem.](http://www.gwern.net/Archiving-URLs)** At the same time, many posts on Less Wrong—including some of the most important posts, which talk about important concepts or otherwise advance our collective knowledge and understanding—are link posts, which means that a **non-trivial chunk of our content is hosted elsewhere**—across a myriad other websites.

If Less Wrong means to be a repository of the rationality community’s canon, we must take seriously the fact that (as [gwern’s research](http://www.gwern.net/Archiving-URLs) indicates) **many or most of those externally-hosted pages will, in a few years, no longer be accessible**.

I’ve taken the liberty of putting together [a quick-and-dirty solution](https://wiki.obormot.net/Archive/LessWrongLinkPostArchive). This is a page that, when loaded, scrapes the external links (i.e., the link-post targets) from the front page of GreaterWrong, and **automatically submits them to [archive.is](http://archive.is/)** (after checking each link to see whether it’s already been submitted). A cronjob that loads the page daily ensures that as new link-posts are posted, they will automatically be captured and submitted to archive.is.

This solution does not currently have any way to scrape and submit links older than those which are on the front page today (2018-09-08). It is also not especially elegant.

It may be advisable to implement automatic link-post archiving as a feature of Less Wrong itself. (Programmatically submitting URLs to archive.is is extremely simple. You send a POST request to `http://archive.is/submit/`, with a single field, `url`, with the URL as its value. The URL of the archived content will then—after some time, as archiving is not instantaneous—be accessible via `http://archive.is/timegate/[the complete original URL]`.)