Harley-ml/lesswrong huggingface dataset:
# A framework for thinking about wireheading

\[Epistemic status: Writing down in words something that isn't very complicated, but is still good to have written down.\]

A great deal of ink has been spilled over the possibility of a sufficiently intelligent AI noticing that whatever it has been positively reinforced for seems to be \*very\* strongly correlated with the floating point value stored in its memory labelled “utility function”, and so through some unauthorized mechanism editing this value and defending it from being edited back in some manner hostile to humans. I'll reason here by analogy with humans, while agreeing that they might not be the best example.

“Headwires” are (given a little thought) not difficult to obtain for humans -- heroin is freely available on the black market, and most humans know that, when delivered into the bloodstream, it generates “reward signal”. Yet most have no desire to try it. Why is this?

Asking any human, they will answer something along the lines of ”becoming addicted to heroin will not help me achieve my goals” ( or some proxy for this: spending all your money and becoming homeless is not very helpful in achieving one's goals for most values of “goals”.) Whatever the effects of heroin, the actual pain and pleasure that the human brains have experienced has led us to become optimizers of very different things, which a state of such poverty is not helpful for.

Reasoning analogously to AI, we would hope that, to avoid this, a superhuman AI trained by some kind of reinforcement learning has the following properties:

1.  While being trained on “human values” (good luck with that!) the AI must not be allowed to hack its own utility function.
2.  Whatever local optima the training process that generated the AI ends up in (perhaps reinforcement learning of some kind) assigns some probability to the AI optimising what we care about.
3.  (most importantly) The AI realizes that trying wireheading will lead it to become an AI which prefers wireheading over aim it currently has, which would be detrimental to this aim.

I think this is an important enough issue that some empirical testing might be needed to shed some light. Item (3) seems to be the most difficult to implement; we in the real world have the benefit of observing the effects of hard drugs on their unfortunate victims and avoiding them ourselves, so a multi-agent environment in which our AI realizes it is in the same situation as other agents looks like a first outline of a way forward here.
# A comment on the IDA-AlphaGoZero metaphor; capabilities versus alignment

Iterated Distillation and Amplification has often been compared to AlphaGo Zero. But in the case of a Go-playing AI, people often don't see a distinction between alignment and capability, instead measuring success on a single axis: ability to win games of Go. But I think a useful analogy can still be drawn that separates out ways in which a Go-playing AI is aligned from ways in which it is capable.

A well-aligned but not very capable Go-playing agent would be an agent that is best modeled as trying to win at Go, as opposed to trying to optimize some other feature of sequence of board positions, but still does not win against moderately skilled opponents. A very capable but poorly aligned Go-playing agent would be an agent that is very good at causing the Go games that it plays to have certain properties other than the agent winning. One way to create such an agent would be to take a value network that rates board positions somewhat arbitrarily, and then use Monte Carlo tree search to find policies that cause the game to progress through board states that are rated highly by the value network.

To a certain extent, this could actually happen to something like AlphaGo Zero. If some bad board positions are mistakenly assigned high values, AlphaGo Zero will use Monte Carlo tree search to search for ways of getting to those board positions (and similarly for ways to avoid good board positions that are mistakenly assigned low values). But it will simultaneously be correcting the mistaken values of these board positions as it notices that its predictions of the values of future board positions that follow from them are biased. Thus the Monte Carlo tree search stage increases both capability and alignment. Meanwhile, the policy network retraining stage should be expected to decrease both capability and alignment, since it is merely approximating the results of the Monte Carlo tree search. The fact that this works shows that the extent to which Monte Carlo tree search increases alignment and capability exceeds the extent to which policy network retraining decreases them.

It seems to me that this is the model that Iterated Distillation and Amplification should be trying to emulate. The amplification stage will both increase capability by increasing the available computing power, and increase alignment because of the human overseer. The distillation stage will decrease both capability and alignment by imperfectly approximating the amplified agent. The important thing is not to drive the extent to which distillation decreases alignment down to zero (which may be impossible), but to ensure that the distillation stage does not cause alignment problems that will not be corrected in the subsequent amplification stage.
# Conceptual problems with utility functions

_\[Epistemic status: strong intuitions that I have had for a long time, but not yet compared with other people's intuitions, and I am not sure how well I can convey my intuitions.\]_

So, a lot of agent foundations research seems to be built on the premise that agenthood is about maximizing the expected value of some utility function, or about having actions that bear some relation to some utility function, or something like that. I think that this premise is wrong except in special cases like where there is only one agent in the world. I don't know exactly how to explain this, so I will just say some stuff.

The Ultimatum Game is a standard example of a game in which there are many Pareto optimal outcomes, and no clear way to choose between them. So if we imagine two "perfectly rational" agents playing the Ultimatum Game against each other, what happens? I think that this question is meaningless. OK, so what happens in real life? People use their notions of fairness to resolve situations like the Ultimatum Game. Fairness is a part of human values, so essentially the answer is that an agent's values are used to resolve the ambiguity of multiple Pareto optimal outcomes.

But wait! In the classical understanding, a utility function is supposed to encode _all_ information about an agent's values. So if there is a notion of fairness relevant to a real-life Ultimatum Game based on monetary payouts, then it supposedly means that the utility function is not just the same as the monetary payouts, and the game is not a true Ultimatum Game at all. But then what is a true Ultimatum Game? Does such a mythical beast even exist? Eliezer had to invent a fairly [far-fetched scenario](https://www.lesswrong.com/posts/HFyWNBnDNEDsDNLrZ/the-true-prisoner-s-dilemma) before he found something that he was willing to call the "true Prisoner's dilemma".

But even in the "true Prisoner's dilemma", the utility function does not appear to capture all of Eliezer's values -- he seems to still be motivated to say that "cooperate" is the right answer based on symmetry and hope, which again appear to be human values. So I propose that in fact there is no "true Prisoner's dilemma", simply because calling something a dilemma is asking you to resolve it with your own values, but your own values are allegedly encapsulated by the utility function which is subject to the dilemma.

I propose instead that agenthood is a robustness to these sorts of games, a sort of continual supply of values which are sufficient to escape from any logical argument purporting to prove that there is no determinate answer as to what action you should take. There is no "pure essence of rational agenthood" that we can simply graft onto the "utility function" of human values to make an aligned AI, because human values are not only a function on world-states to be learned but also a part of the decisionmaking process itself. This seems to suggest radically different approaches to the alignment problem than what is present so far, but I am not sure what they are yet.
# Agents That Learn From Human Behavior Can't Learn Human Values That Humans Haven't Learned Yet

_\[Epistemic status: ¯\\_(ツ)_/¯ \]_

  

[Armstrong and Mindermann](https://arxiv.org/abs/1712.05812) write about a no free lunch theorem for inverse reinforcement learning (IRL): the same action can reflect many different combinations of values and (irrational) planning algorithms.

I think even assuming humans were fully rational expected utility maximizers, there would be an important underdetermination problem with IRL and with all other approaches that infer human preferences from their actual behavior. This is probably obvious if and only if it's correct, and I don't know if any non-straw people disagree, but I'll expand on it anyway.

Consider two rational expected utility maximizing humans, Alice and Bob.

Alice is, herself, a value learner. She wants to maximize her true utility function, but she doesn't know what it is, so in practice she uses a probability distribution over several possible utility functions to decide how to act.

If Alice received further information (from a moral philosopher, maybe), she'd start maximizing a specific one of those utility functions instead. But we'll assume that her information stays the same while her utility function is being inferred, and she's not doing anything to get more; perhaps she's not in a position to.

Bob, on the other hand, isn't a value learner. He knows what his utility function is: it's a weighted sum of the same several utility functions. The relative weights in this mix happen to be identical to Alice's relative probabilities.

Alice and Bob will act the same. They'll maximize the same linear combination of utility functions, for different reasons. But if you could find out more than Alice knows about her true utility function, then you'd act differently if you wanted to truly help Alice than if you wanted to truly help Bob.

So in some cases, it's not enough to look at how humans behave. Humans are Alice on some points and Bob on some points. Figuring out details will require explicitly addressing human moral uncertainty.
# Decision-theoretic problems and Theories; An (Incomplete) comparative list

So, this is a work-in-progress, but the idea is to a) get an eventually exhaustive list of decision-theoretic problems, and b) detail the '''answers''' given by each major decision theory.

(Stretch goals would including listing other kinds of properties, and representing disagreement about the ''answers'')

Proposed amendments and additions are welcomed!
# LW Update 2018-7-14 – Styling Rework, CommentsItem,
Performance

**Performance Updates**

*   Implemented server side rendering for the frontpage, and for post pages. Now, most relevant content should load quite quickly (so you can begin reading), and then the bulk of the web-app will load a few seconds later.
*   Cleaned up some extraneous code that adding to our overall load times (in particular for LaTeX heavy posts)
*   Temporarily disabled some hover-over tooltips on the CommentsItem, which was slowing down rendering of pages with lots of comments.

**Legacy RSS Feeds**

*   If you were subscribed to old LW via RSS, your subscription should work again (pointing to the All Posts feed). This currently includes all posts including spam, but in the near future should be limited to posts with N karma (probably 10), which should resolve that.

(Thanks to use jimrandomh who helped out a bunch with performance and RSS updates)

**Nested Frontpage Comments**

*   Open Source contributor ForrestWeiswolf submitted a PR that allows comments on the frontpage to be properly nested, while still allowing you to click-to-see-parent. This should make the Recent Discussion section a bit easier to parse.
*   Relatedly, on a Post item on the frontpage, if you click the comments icon, you will now see the most recent 5 comments, properly nested, _with_ the ability to show additional parents. This should make the "show recent comments" button much more useful.

**Notifications**

*   Fixed a bug where if you had never clicked the notification bell, it wouldn't light up when you got notifications.

**Moderator Sidebar**

*   Various small tweaks to make this more useful to the sunshine squad.

**AlignmentForum**

*   There is now a subforum of LessWrong available at AlignmentForum.org, currently in Open Beta. More details [here](https://www.lesswrong.com/posts/JiMAMNAb55Qq24nES/announcing-alignmentforum-org-beta).
*   AlignmentForum posts on LessWrong will now display a second type of karma (roughly representing "karma by technical alignment researchers.")

**Styling Rework**

*   Since the AlignmentForum and some other upcoming projects require custom themes for LessWrong, and our existing stylesheets were a bit messy, we took the opportunity to refactor much of our styling code. We are now moving towards JSS, which a) consolidates theming decisions into a single file, which b) has full JS programmability.

**Code Blocks**

*   Code blocks in posts now display properly.

Thanks to new contributors berekuk, compwron and adrusi, who contributed several bugfixes, including code blocks properly displaying on posts.
# Meetup Cookbook

My spouse and I have been running LessWrong meetups in two different cities for the past six years. Over time, we've gotten lazier and lazier about organizing, while still maintaining similar results, mostly by coming up with a bunch of simple recipes and scripts for running meetups. Here I have documented how we do it in excruciating detail, so others can use what we do.
# Announcement: AI alignment prize round 3 winners and next round

We (Zvi Mowshowitz and Vladimir Slepnev) are happy to announce the results of the third round of the [AI Alignment Prize](https://www.lesswrong.com/posts/SSEyiHaACSYDHcYZz/announcement-ai-alignment-prize-round-2-winners-and-next), funded by Paul Christiano. From April 15 to June 30 we received entries from 12 participants, and are awarding $10,000 to two winners.

We are also announcing the fourth round of the prize, which will run until December 31 of this year under slightly different rules. More details below.

### The winners

First prize of $7,500 goes to Vanessa Kosoy for [The Learning-Theoretic AI Alignment Research Agenda](https://agentfoundations.org/item?id=1816). We feel this is much more accessible than previous writing on this topic, and gives a lot of promising ideas for future research. Most importantly, it explains why she is working on the problems she’s working on, in concrete enough ways to encourage productive debate and disagreement.

Second prize of $2,500 goes to Alexander Turner for the posts [Worrying About the Vase: Whitelisting](https://www.lesswrong.com/posts/H7KB44oKoSjSCkpzL/worrying-about-the-vase-whitelisting) and [Overcoming Clinginess in Impact Measures](https://www.lesswrong.com/posts/DvmhXysefEyEvXuXS/overcoming-clinginess-in-impact-measures). We are especially happy with the amount of good discussion these posts generated.

We will contact each winner by email to arrange transfer of money. Many thanks to everyone else who sent in their work!

### The next round

We are now announcing the fourth round of the AI Alignment Prize. Due the drop in number of entries, we feel that 2.5 months might be too short, so this round will run until end of this year.

We are looking for technical, philosophical and strategic ideas for AI alignment, posted publicly between July 15 and December 31, 2018. You can submit links to entries by leaving a comment below, or by email to apply@ai-alignment.com. We will try to give feedback on all early entries to allow improvement. Another change from previous rounds is that we ask each participant to submit only one entry (though possibly in multiple parts), rather than a list of several entries on different topics.

The minimum prize pool will again be $10,000, with a minimum first prize of $5,000.

Thank you!
# September CFAR Workshop

Applications are open for the last general-admission CFAR workshop of the year. We'll be taking about 25 people total. Interviews are open from now until 8/14, at which point we'll send out acceptance/waitlist emails

Apply at [cfaradmissions.youcanbook.me](http://cfaradmissions.youcanbook.me)

Tickets are $3900, with limited financial aid.
# Generalized Kelly betting

\[tl;dr: It's a mess, don't go there\]

\[Thanks to Diff for proof reading this post\]

[Kelly betting](https://en.wikipedia.org/wiki/Kelly_criterion) has the [very nice property](http://shlegeris.com/2018/04/11/kelly) that if a gambler is betting according to a given world model, and the amount of money the gambler starts out with equals to the prior probability of that model, then after each round of bets, this gamblers money will equal the current posterior probability.

The problem with Kelly betting is that it relies on only being given one bet at a time, and that the previous bet will be evaluated before you are asked to bet on a new question. Compare this to the situation faced by the traders in a Logical Inductor, where there are always multiple bets every round and the traders don't know when any bet will be settled.

I have (almost) calculated the generalized [Kelly criterion](https://en.wikipedia.org/wiki/Kelly_criterion), in the case of two dual outcome simultaneous bets, with general market odds and general gambler beliefs. The only remaining part of this calculation is a cubic equation.

**Send me an e-mail (linda.linsefors@gmail.com) if you want my notes.** There is no guarantee that I will read all blog post comments.

* * *

Solving this last equation for general market odds, is in principle not very hard. You can find the [general solution to cubic equations on Wikipedia](https://en.wikipedia.org/wiki/Cubic_function#General_solution_to_the_cubic_equation_with_real_coefficients). Except in this specific case the equation is:

< refresh to render LaTeX > < refresh to render LaTeX >

< refresh to render LaTeX >

solve for < refresh to render LaTeX > .

The probability that I will get this right by hand is near zero. Maybe someone with Mathematica, or a similar tool could help me?

For the notation, there are two simultaneous and independent bets. Each bet has two outcomes, denoted by index < refresh to render LaTeX > .

< refresh to render LaTeX >

And the ' superscript denotes the second bet. By definition

< refresh to render LaTeX >

< refresh to render LaTeX > is just a help variable I made up. It does not have a super clear interpretation, but just happens to be the key to calculating everything else.

* * *

As mentioned above, I don't have the final formula for the generalized Kelly criterion, in the case of two dual outcome simultaneous bets, with completely general market odds and gambler beliefs, not until someone solves the above equation. What I currently do have is the special case where the market probabilities for both statements is 50%, and this special case already shows that the nice property of money representing probability, **is not preserved**.

The Bayesian updating factor for hypotheses < refresh to render LaTeX > , given two independent observations < refresh to render LaTeX > and < refresh to render LaTeX > , and < refresh to render LaTeX > , should be

< refresh to render LaTeX >

The update factor for the amount of money that a gambler following hypothesis < refresh to render LaTeX > has, given the above circumstances, is

< refresh to render LaTeX >
# LW Update 2018-07-18 – AlignmentForum Bug Fixes

Git commit: 882df0fe53f3c2bb04fac76d5b631f32cab28b1b

**AlignmentForum**

*   If a non-AlignmentForum member's post gets moved to AlignmentForum, they can now comment on that post.
*   Fixed the bug on LessWrong where canceling a vote on a non-AF post briefly showed a flicker of AF karma.

**Stickies**

*   Previously we only had a single "sticky" option for posts, which meant it was slightly awkward when we wanted to sticky things in either the Meta forum, or AlignmentForum. Meta and AF now have their own sticky setting.

**Form Group Styling**

*   Form Groups (such as editing your user profile, or the moderation settings for high karma users) are a little easier to parse now.

There have been a few more performance improvements for long comment threads.
# Solving the AI Race Finalists

[Good AI](https://www.goodai.com/) offered a prize for writing about AI races. The results are in and here are the winners:

#### **Top scoring solutions ($3,000 each)**

*   **Kesavan Athimoolam,** [Solving the Artificial Intelligence Race: Mitigating the problems associated with the AI Race](http://mirror.goodai.com/judges/Solving_the_AI_race_KA.pdf)
*   **Alexey Turchin and & David Denkenberger,** [Classification of Global Solutions for the AI Safety Problem](http://mirror.goodai.com/judges/TurchinDenkenberger__Global_Solutions_of_the_Artificial_Intelligence_Safety_Problem_17.pdf)
*   **Ehrik L. Aldana,** [A Theory of International AI Coordination: Strategic implications of perceived benefits, harms,capacities, and distribution in AI development](http://mirror.goodai.com/judges/Aldana_Ehrik.pdf)

#### **Runners-up ($2,000 each)**

*   **David Klimek,** [Framework for managing risks related to emergence of AI/AGI](http://mirror.goodai.com/judges/David_Klimek.pdf)
*   **Gordon Worley,** [Avoiding AGI Races Through Self-Regulation](https://mapandterritory.org/avoiding-ai-races-through-self-regulation-1b815fca6b06)
*   **Morris Stuttard & Anastasia Slabukho,** [The AI Engineers’ Guild: proposal for an AI risk mitigation strategy](http://mirror.goodai.com/judges/AI_RACE_THE_AI_ENGINEERS_GUILD_by_M_Stuttard_A_Slabukho.pdf)
# Conceptual problems with utility functions, second attempt at explaining


_\[Followup to: [Conceptual problems with utility functions](https://www.lesswrong.com/posts/Nx4DsTpMaoTiTp4RQ/conceptual-problems-with-utility-functions)\]_

The previous post was me flailing my arms about wildly trying to gesture at something. After some conversations I think I have a somewhat better idea of what it is and can try to explain it somewhat more systematically. Let us see if it works.

The point that I want to make clear I agree with, is that there is a difference between "valuing fairness" (for example) because you want everyone to get something, versus "valuing fairness" for tactical reasons. For example, suppose that you are playing the Ultimatum Game where the point is to divide the profit from a joint project, and each person thinks they put in more effort than the other and so deserves a larger share. Then maybe you think that the "fair outcome" is for you to get 60%, while the other person thinks that the "fair outcome" is for you to get 40%.

I think a reasonable way to deal with this scenario is: each of you plays a mixed strategy with the end result that on average, both players will get 40% (meaning that 20% of the time the players will fail to reach agreement and nobody will get anything). The nice things about this strategy are (A) it gives both players at least some money and (B) it satisfies a "meta-fairness" criterion that agents with more biased notions of fairness aren't able to exploit the system to get more money out of it.

So now in our decisionmaking process there are two notions of "fairness": the initial notion of fairness that made us decide that the fair split is 60/40, and the "meta-fairness" or "inexploitability" that made us sacrifice some of the other person's money so that we prefer 40/40 to 40/60. Certainly these are different. But my point is that they are not as different as they appear, and I am not sure that there is a principled way of separating your decisionmaking process into two components "utility function" and "decision theory". Moreover, even if there is a clean way of making such a separation, it is not clear to me that "meta-fairness" is a much simpler concept than "object-level fairness". And it seems to me that the utility functions/game theory framework presumes not only that it is a simpler concept, but that it is somehow conceptually equivalent to the concept of "maximization" (which doesn't seem to be true at all).

(The example might seem deceptively simple in mathematical terms, but the difficulty in formalizing it in a utility functions framework is that you are comparing different possible worlds in which your opponent has different utility functions, and you are measuring inexploitability in terms of how much resources they take, not how their utility function is affected. So you at least need the concept of a "resource".)
# San Francisco Meetup: Articles

We'll be discussing [Problem Solving with Mazes and Crayon](https://www.lesswrong.com/posts/CPBmbgYZpsGqkiz2R/problem-solving-with-mazes-and-crayon) and [Book Review: Capital in the Twenty-First Century](http://slatestarcodex.com/2018/06/24/book-review-capital-in-the-twenty-first-century/). We'd prefer it if you read the articles beforehand, but there will be printed copies available.

For help getting into the building, please call (or text, with a likely-somewhat-slower response rate): 203-503-7508 (general meetup info at 301-458-0764).

**Format:**

We meet and start hanging out at 6:15, but don’t officially start doing the meetup topic until 6:45-7:00 to accommodate stragglers. Usually there is a food order that goes out before we start the meetup topic.

**About these meetups:**

The mission of the SF LessWrong meetup is to provide a fun, low-key social space with some structured interaction, where new and non-new community members can mingle and have interesting conversations. Everyone is welcome.

We explicitly encourage people to split off from the main conversation or diverge from the topic if that would be more fun for them (moving side conversations into a separate part of the space if appropriate). Meetup topics are here as a tool to facilitate fun interaction, and we certainly don’t want them to inhibit it.
# 12 Virtues of Rationality posters/icons

I was trying to take a break today, and ended up redesigning the staircase at our group house (Event Horizon), which is the first thing you see when you enter the house. One thing me and Claire ([mingyuan](https://www.lesswrong.com/users/mingyuan)) came up with, was to design small posters for each of the twelve virtues of rationality and line them up on the walls. And I figured I would also let other people have access to the posters, in case you want to do something similar. Here are all the 12 icons/posters. The general idea was inspired by these awesome [minimalist philosophy posters.](http://studiocarreras.com/philographics/)

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236016/assembly-12_fxgqtk.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236018/assembly-02_p43knj.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236018/assembly-03_huqtv7.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236018/assembly-04_syrz2n.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236019/assembly-05_hfm8lk.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236018/assembly-06_s03zqx.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236016/assembly-07_eub4ba.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236016/assembly-08_vjuyvr.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236016/assembly-09_hjxsne.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236016/assembly-10_wdgmby.svg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236016/assembly-11_afl4hf.svg)

And I've uploaded the illustrator file I've used [here](https://drive.google.com/file/d/1s6HQTyFPQJoUCL-2H_jq3tGTdSBuM1k6/view?usp=sharing).

The twelfth virtue doesn't have an icon, we chose to represent it with an empty frame.

Here is the staircase after we finished putting everything up (sorry, lighting isn't perfect):

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236573/IMG_0169_sslo9a.jpg)

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1532236626/IMG_0168_r3k2j2.jpg)
# Who Wants The Job?

Over this last month, my wife and I searched for and hired a new nanny, as ours had decided to learn programming and move to The Bay.

We ended up with a wonderful woman we found through a personal recommendation on a local mailing list. Her previous employer posted saying, hire this woman, she’s fantastic, and this was indeed the case.

Before we realized we’d found her, various people encouraged us to post a job listing on a website, so we did so on Sittercity.

That went… less well.

We got over one hundred applications.

The majority of them had profiles that did not match the requirements of the job. About half were only available for a few months or for part time work. Many others wouldn’t work with multiple children, or had higher salary requirements than we listed.

The vast majority of them had major spelling and/or grammar errors in their profiles. Also in their messages to us.

From the remaining profiles, we reached out to a few dozen applicants.

The majority of them _did not respond to us at all. _

Of those who did respond, several _did not answer the phone for the initial interview _and provided no explanation.

Of those who did answer, several _were actively rude on the phone. _

Of those who were not rude on the phone, several _did not engage with the questions being asked or show any interest in the job,_

Of those who passed that screening and were asked to come for an in person interview, _fully half of them failed to show up, most with no warning or cancellation. _In all such cases, they didn’t contact us again. In other cases, they cancelled, but failed to make any real effort to reschedule.

As a result of all this, _we only ended up doing two in person interviews. _Because it turns out that getting people to show up, at all, is super hard. One of them seemed acceptable in a punch. The other we hired.

The vast majority of people who were on a job site, seeking a job, were not capable of tasks like: Write a profile page in English without major mistakes. Respond promptly when an employer contacts you to respond to your application. Talk politely on the phone and sound like you are listening and care about the job. Show up to your interview.

Yes. Standards are _that low. _

You are much more employable than you think.

If you’re wondering why employers say it’s so hard to hire people despite getting a hundred applications for every position? That’s why.

If you’re wondering why many people can’t find work? I can’t help but wonder if it’s because they can’t do the very basic things even at high leverage points like the interview process. Things like showing up and responding to emails. Being on time. Being polite. Making sure the profile you show the world doesn’t have major errors and matches the jobs you’re applying to. Acting like you actually want the job.

Those are standards that _98% or so _of applications we got failed to live up to. Presumably the same people, failing to live up to them, apply again and again, failing those standards again and again, wondering why they can’t get hired.
# The Biggest Problem in your Life

Mathematician Richard Hamming used to ask scientists in other fields "What are the most important problems in your field?" partly so he could troll them by asking "Why aren't you working on them?" and partly because getting asked this question is really useful for focusing people's attention on what matters. CFAR developed the technique of "Hamming Questions" as different prompts to get your brain to (actually) think about the biggest problems, bottlenecks, and unspoken desires in _your_ life.

Plan for the meetup:

1\. I will go through a list of prompts to get you to think of the various bugs in your life and the underlying "Hamming problems" that lie at the core of what's preventing you from being all you want to be.

2\. We will break into pairs, and each person will describe their Hamming problems and why they haven't solved them yet to another person, while looking in their eyes. The other person doesn't have to hear what's being said (I'll provide ear plugs for those who want them). The goal of saying this out loud is to let yourself be vulnerable and make the problem real in your attention.

3\. If enough people wish to discuss their Hamming problems, we will break into small groups where one person will present their problem and (hopefully) receive compassion, clarification, and feedback from the others.

These exercises will likely be somewhat outside the comfort zone for most of us, and are partly intended to stretch that comfort zone. You are very welcome to drop out after part 1 (which will take 30-45 by itself) or part 2; Luna has many lovely spaces to hang out in. However, if you choose to participate in any of the parts, please be committed and fully present. Also, please be on time. The success of the exercise depends a lot on the atmosphere created in the space.

Please bring:

1\. Something to write with/on, preferably paper-based and not electronic.

2\. Ear plugs, in case I misjudge the total number needed or you prefer your own.

3\. An open and introspective mindset.

Thank you to the Lunatics for graciously hosting this event at Luna Labs!

(For more details and exact location, please join our [Google group](https://groups.google.com/forum/#!forum/overcomingbiasnyc)).
# Let's Discuss Functional Decision Theory

I've just finished reading through [Functional Decision Theory: A New Theory of Rationality](https://arxiv.org/pdf/1710.05060.pdf), but there are some rather basic questions that are left unanswered since it focused on comparing it to Casual Decision Theory and Evidential Decision Theory:

*   How is Functional Decision Theory different from Timeless Decision Theory? All I can gather is that FDT intervenes on the mathematical function, rather than on the agent. What problems does it solve that TDT can't? (Apparently it solves Mechanical Blackmail with an imperfect predictor and so it should also be able to solve Counterfactual Mugging?)
*   How is it different from [Updateless decision theory](https://wiki.lesswrong.com/wiki/Updateless_decision_theory)? What's the simplest problem in which they give different results?
*   Functional Decision Theory seems to require counterpossibilities, where we imagine that a function output a result that is different from what it outputs. It further says that this is a problem that isn't yet solved. What approaches have been tried so far? Further, what are some key problems within this space?
# LessWrong Meetup for Hamming Circle's

While he was at Bell Labs, the mathematician Richard Hamming was known ([http://www.paulgraham.com/hamming.html](https://l.facebook.com/l.php?u=http%3A%2F%2Fwww.paulgraham.com%2Fhamming.html&h=AT2Laz2cBlPJioHzjsThR6Ofwj2OtBKDkHugXJQp2F0UVFGDGJukeUMFmLMw-e8vl6RfXIpJpg9WURz2z_19kNOFOlKhP38vVUYhVMCfItz9iVpVokBKVDlzOICYv1w24DFed5hiJncaLMWaPA)) for sitting with different prestigious scientists and asking: “What are the most important problems in your field?” A week later, he’ll follow up with asking,”What important problems are you working on?”  

Kind of an arse, but he gets to a really critical point. Some problems are much more important than others. The same question can be applied to your personal life: “what are the important problems in your life, and what is stopping you from working on them?”  

For this meetup, we'll try to use a group rationality technique called a “Hamming Circle”, where we cooperatively help each other identify and try to resolve bottlenecks to what we each consider to be the most important problems in our lives. This can potentially be very useful.  

13:00-13:30 Arriving and chatting  
13:30-15:00 First Workshop Block  
13:00-15:45 Mingling  
15:45 -17:15 Second Workshop Block  
17:15 -18:00 Mingling
# ISO: Name of Problem

I'm looking for a name for a problem. I expect it already has one, but I don't know what it is.

The problem: suppose we have an AI trying to learn what people want - e.g. an IRL variant. Intuitively speaking, we point at a bunch of humans and say “figure out what they want, then do that”. A few possible ways the AI could respond:

*   “Hmm, to the extent that those things have utility functions, it looks like they want friendship, challenge, status, etc…”
*   “Hmm, it looks like they want to maximize the number of copies of the information-carrying molecules in their cells.”
*   “Hmm, it looks like they’re trying to maximize entropy in the universe.”
*   “Hmm, it looks like they’re trying to minimize [physical action](https://en.wikipedia.org/wiki/Action_(physics)).”

Why would the AI think these things? Well, you’re pointing at a bunch of atoms, and the microscopic laws of motion which govern those atoms can be interpreted as minimizing a quantity called action. Or you’re pointing at a bunch of organisms subject to a selection process which (locally) maximizes the number of copies of some information-carrying molecules. How is the AI supposed to know which optimization process you’re pointing to? How can it know which level of abstraction you’re talking about?

What data could tell the AI that you're pointing at humans, not the atoms they're made of?

This sounds like a question which would already have a name, so if anybody could point me to that name, I'd appreciate it.
# The Evil Genie Puzzle

Foolish you. You decided to rub a lamp in the hope that a genie would appear and you found one, only it's an Evil Genie. He tells you only have two options: either wishing for your perfect life or being pelted with rotten eggs. You start thinking about how wonderful your new life will be, but of course, there's a catch.

In the past, wishing for a perfect life would have come with side-effects attached, but due to past abuses, the Council of Genies has created new, updated rules which ban any unwanted side-effects for the person who makes the wish or any of their loved one's. However, there's no rule against messing with other people. He tells you that he has already predicted your choice and if he predicted that you will wish for a perfect life, he has already created a million clones of you in identical rooms on Mars who will believe that they've just rubbed the lamp. However, since they've only hallucinated this, he doesn't have any obligations towards them and he will use his powers to torture them if they will choose the perfect life. On the other hand, if he predicts that you will choose to be pelted with rotten eggs, then he won't create any clones at all. Genies are well known to be perfect predictors.

Assuming you are a perfectly selfish agent who doesn't care about their clones, what decision ought you take? If you choose to be pelted, there's a 100% chance that if you had wished for a perfect life, you would have received it. On the other hand, if you wish for a perfect life, there's an overwhelming chance that you will wish that you had wished to be pelted. No matter what decision you make, it seems that you will immediately regret having made it (at least before it is revealed whether you are the original in the case where you choose the perfect life).

**Extra Info:** Some people might argue that perfect clones necessarily are you, but they don't actually have to be perfect clones. The genie could just clone someone else and give them the memory of finding the lamp and this ought to create the requisite doubt as to whether you are a real human on Earth or a clone on Mars.

Prior version of this punished the clones for existing or if it is predicted that they chose the perfect life, but the clones are now punished if they choose the perfect life.
# Invest In Yourself - Los Angeles LW/SSC Meetup #68 (Wednesday, July 25th)

Location: Wine Bar next to the Landmark Theater in the Westside Pavilion (10850 W Pico Blvd #312, Los Angeles, CA 90064).  We will move **upstairs (to the 3rd floor hallway) as soon as we reach capacity**.

Time: 7 pm (July 25th)

Parking: Available in the parking lot for the entire complex.  **The first three (3) hours are free and do not require validation** (the website is unclear and poorly written, but it may be the case that if you validate your ticket and leave before three hours have passed, you will be charged $3)**.  After that, parking is $3 for up to the fifth (5) hour, with validation.**

Contact: The best way to contact me (or anybody else who is attending the meetup) is through our **Discord**.  Feel free to message me (T3t) directly.  Invitation link: [https://discord.gg/TaYjsvN](https://www.google.com/url?q=https%3A%2F%2Fdiscord.gg%2FTaYjsvN&sa=D&sntz=1&usg=AFQjCNGp_tNSuquq0qHTGPUXIO3mz0dTNw)

Topic: Our discussion this week will be on where to preferentially focus your time, energy, and resources.  Thanks to Frankie for the suggestion which inspired the topic; optional reading: http://urbanhonking.com/kmikeym/
# Would you benefit from audio versions of posts?

I know two people who have a hard time reading, but totally enjoy audio versions of blogposts and were only able to consume the sequences via audiobook.

Curious how common this is, and how much low hanging fruit there is vis-a-vis either making audio versions of posts easily accessible, or finding a high quality text-to-speech thing to provide it by default.
# LW Update 2018-07-27 – Sharing Drafts

Most of our current work continues to be on making the codebase theme-able, so that [AlignmentForum](http://alignmentforum.org/) and the upcoming new [EA Forum](http://effective-altruism.com/ea/1qv/ea_forum_20_initial_announcement/) can easily be re-styled.

However, we've got a new feature that I'm hoping will (eventually) play a significant role in changing how people use LessWrong: sharing drafts.

When you create a new post, you can now share it with specific users. And then, if you submit it as a draft, they can view it even though it's not posted publicly. (They can also comment on it. It's up to you whether you want to encourage this for a given post)

Currently there are no notifications associated with this, and will probably be awhile before we build out additional features one might want for this (easily sharing with groups, in-line comments, allowing multiple users to edit, version control, etc). But I wanted to at least get this basic feature out sooner rather than later, and am hoping for the longterm for this to be part of the move towards a more peer-review-esque world.

Right now LessWrong sort of encourages you to come up with an idea, write it up, post it, and then maybe there are comments and critiques but you don't actually go back to rework the original post.

Longterm I'm excited by the prospect of "write the post" just being the beginning, and having various formal and informal systems for having _improve_ that post being a continuous, important part of the process.
# Algo trading is a central example of AI risk

I suspect this observation is far from original, but I didn't see it explicitly said anywhere, so it seemed worthwhile spelling it out.

The paperclip maximizer is a popular example of unfriendly AI, however it's not the most realistic one (obviously it wasn't meant to be). It might be useful to think which applications of AI _are_ the most realistic examples, i.e. which applications are both likely to use state-of-the-art AI and are especially prone to failure modes (which is not to say that other applications are _not_ dangerous). In particular, if AI risk considerations ever make it into policy, such an analysis is one thing that might help to inform it.

One application that stands out is algorithmic trading. Considering the following:

*   Algorithmic trading is obviously lucrative and has strong economic incentives encouraging it.
*   Algorithmic trading has some aspects that are zero-sum_,_ leading to an especially vicious technological race. In this race, there is no natural stopping point of "good enough": the more powerful your algorithm, the better.
*   Even ignoring the possibility of competing AIs, there is no "good enough" point: acquiring more money is a goal that is either unlimited or, if it _has_ a limit then this limit already requires enough power for a [pivotal event](https://arbital.com/p/pivotal/).
*   The domain is such that it would be very advantageous for your AI to build detailed models of the world as a whole (at least the human world) and understand how to control it, including in terms of human psychology, economics and technological development. These capabilities are precisely what a pivotal event would require.
*   Algorithmic trading doesn't require anything close to AGI in order to start paying off. Indeed, it is already a very active domain. This means that transition from subhuman to superhuman intelligence is more likely to happen in a way that is unintended and unplanned, as the algorithm is gradually scaled up, whether in terms of computing power or in other ways.
*   Last but definitely not least, the utility function is exceptionally simple. Formally specifying a "paperclip" might still be complicated, but here we only need something like "the amount of money in a given bank account". This means that this application requires almost nothing in the way of robust and transparent learning algorithms: sufficiently powerful straightforward reinforcement learning might do absolutely fine. Because of this, an algorithmic trading AI might lack even those safety mechanisms that other applications would require _before_ scaling up to superintelligence.
# Pretense

There's a kind of yearning, to be _that_ person who can do _those_ things - this is self-actualization, yet corrupted. I often feel pulled in this way. I find myself wanting to be a certain person _now_, to be producing and being and feeling that way _now_, and I catch myself acting, speaking, signalling as if I were there _now_. As if I could make people happy by tiling the universe with smiley-faces.

There's a revulsion that comes with this, for me - the sense of wearing a heavy coat, of playing a role, of acting instead of connecting. At times, there _is_ a desire to connect: I begin to speak earnestly, but then comes indecision, a "social acceptance" reflex blunting my emotions and diluting my speech. And then, pain, regret, and shame. Even now, it looms: Can I even post this?

There's a certain lightness I catch and wield, from time to time. The glimmer of a fresh idea, the flow of words straight from the heart through the fingertips, the carefree, liberating simplicity of dropping pretense in a conversation. Why is this so rare?

I feel now the staggering weight of a day's trivialities - the subconscious obeisances paid to circumstance and habit, the pretentious acting out of cached responses, the molding of personality to meet the Past's expectations. There are massive costs - in time, in experience, in moments.

And yet, focusing on "being oneself", one comes to fret whether they're doing it right, or enough, or too much. Self-consciousness takes over, and back on goes the coat.

After CFAR, there was a precious week when I channeled myself during these moments. Pain did not clear its desk for joy, but I paid attention. My life beat to a satisfied rhythm. I felt no urge towards trivial things, towards pretense.

Weeks passed, and I slowly forgot. My experience of the workshop faded into a collection of moments: quietly gazing at the furious red skyline; laughing and singing despite the iceberg inching closer; hearing, and being heard. Ever-so-close bonds loosened, and then crumbled a bit, their essence blown away into the now-yawning distances between us.

I want to take off the coat.