Harley-ml/lesswrong huggingface dataset:
# Leaving beta: Voting on moving to LessWrong.com

It took longer than we hoped, but LessWrong 2.0 is finally ready to come out of beta. As [discussed in the original announcement](http://lesswrong.com/lw/pfl/lw_20_open_beta_live/), we’re going to have a vote on whether or not to migrate the new site to the lesswrong.com URL. The vote will be open to people who had 1,000 or more LW karma at the time we announced the vote back in September, and they’ll receive a link by email or private message on the current LessWrong.com. If you had above 1000 karma in September and did not receive an email or PM, send an email to habryka@lesserwrong.com and we will send you the form link.

_If we move, we will **not** break old incoming links to LessWrong.com._

We take rationalist virtues seriously, and I think it’s important that the community actually be able to look at the new implementation and vision and be able to say “no thanks.” If over half of the votes are to not migrate, the migration will not happen and we’ll figure out how we want to move forward with the website we’ve built.

Unfortunately, the alternative option for what will happen with the lesswrong.com URL is not great. Before I got involved, the original dominant plan was to replace it with a static HTML site, which would require minimal maintenance while preserving the value of old Sequences articles. So in the absence of another team putting forward heroic efforts and coordinating with Trike, MIRI, etc. that would be the world we would be moving towards.

**Why not just keep things as they are?** At the time, it was the consensus among old regulars that LW felt like an abandoned ghost town. A major concern about keeping it alive for the people still using it was that newcomers would read Sequences articles linked from elsewhere, check out the recent discussion and find it disappointing, and then bounce off of LW. This reduced its value for bringing people into the community.

More recently, various security concerns have made it a worse option to just keep old websites running – Trike has run into some issues where updating the server and antiquated codebase to handle security patches proved difficult, and they would prefer to no longer be responsible for maintaining the old website.

**In case you’re just tuning in now, some basic details:** I’ve been posting on LW for a long time, and about two years ago thought I was the person who cared most about making sure LW stayed alive, so decided to put effort into making sure that happened. But while I have some skills as a writer and a programmer, I’m not a webdev and not great at project management, and so things have been rather slow. My current role is mostly in being something like the ‘senior rationalist’ on the team, and supporting the team with my models of what should happen and why. The actual work is being done by a combination of Oliver Habryka, Raymond Arnold, and Ben Pace, and their contributions are why we finally have a site that’s ready to come out of beta.

You can read more about our vision for the new LessWrong [here](https://www.lesserwrong.com/posts/HJDbyFFKf72F52edp/welcome-to-lesswrong-2-0).
# Appropriateness of Discussing Rationalist Discourse of a Political Nature on LW?

There was recently a back-and-forth between Slate Star Codex and Nathan Robinson of _Current Affairs_, a major national news magazine on the political left in the United States. It didn't end well. I think it would serve as a good example in how I think rationalist diaspora members could better think and go about publicly engaging those outside the community on topics of common interest. I wouldn't publish it without running it by Scott first. But I don't want to waste the time to write a draft if it wouldn't be appropriate content for LW anyway.

I could post the write-up on my own blog and submit it as a community post. It wouldn't have any relevance to people outside the rationalist diaspora, so I'd prefer to post it to LW, but my own blog would be fine. If submitting it as a link/community post would be frowned up as well, that'd be fine with me too. I just want to know what the expected norms are here. If the answer to these questions would depend on the content of my write-up, that's also fine. I can drum something up, come back here, and then get feedback. Anyway, if the moderators or anyone else wants to give me their two cents, that'd be great.
# Looking and the no-self

_(This post is a reply to Kaj's ["My attempt to explain Looking"](https://www.lesserwrong.com/posts/mELQFMi9egPn5EAjK/my-attempt-to-explain-looking-insight-meditation-and) and Valentine's [Kenshō](https://www.lesserwrong.com/posts/tMhEv28KJYWsu6Wdo/kensh). The target audience is people who feel like they have a good grasp on the underlying concepts but of course the post might also be interesting for other people)_

I have the feeling that both you and Valentine mix the concept of Looking together with no-self even through they are independent for each other. In the Zen Buddhist theoretical landscape they are linked together but linking them together is a specific position of that tradition.

My main approach and the concept I have to think about this space comes from Danis Bois. He is an interesting Frenchman, who discovered mediation while learning to be an osteopath. The next ten years he spent 8 hours per day meditating, traveled to guru's in India and taught what he called at the time Fascia therapy. At a time where he was successful with his life in the spiritual community he thought they were too dogmatic and decided to go to university. He studied education as his major and become a professor in Portugal and recently retired from being a professor.

There's no renunciation of the self in Bois work. He grants that it often makes sense to think about having multiple 'I' but neither of  them is false. On the other hand having a strong feeling of existence is a key in his way.

David Chapman wrote [a lot](https://vividness.live/category/buddhism/consensus-buddhism/) about the conflict between Consensus Buddhism and Tantra Buddhism. Zen Buddhism is part of what Chapman calls consensus Buddhism. One of the key differences is that consensus Buddhism seeks the renunciation of the self while Tantra doesn't.

While the path of renunciation exists, I consider it to be in conflict with the standard rational goals of having an impact on the world. Given the predominance of Zen in popular culture it might seem like it's the only valid path, but it's not the only spiritual path that enables the ability to Look.

Both Danis Bois paradigm and Tantra care a lot about relationships and consider relating to be important. Going deeply into what that means is a subject for another post but fortunately we have the shared concept of Circling. Circling is about exploring how we relate to each other.

Just like mindfulness meditation is a condensed version of Zen, Circling feels to me like a condensed version of a spiritual path of being in relation.

In my model of the world I can imagine a person who spends a lot of time Circling developing the ability to look in the process. At the same time I wouldn't expect them to renunciation their self.
# Request for "Tests" for the MIRI Research Guide

Lately I've been looking into learning the material in the MIRI research guide. After some time, I noticed that my biggest trepidation about diving in was the nagging question of, "But how will I know if I actually learned the stuff?"

Once I realized that was the thing holding me back, it became easier to think about solutions. Some of the topics correspond with courses that are common in universities, so I can pilfer final exams from their sites (woot to MIT open courseware). But for other topics I wanted to see what people here thought.

In whatever domain you specialize in, what are some examples of problems or questions that one can only answer by having a solid understanding of a huge swath of said domain? Below I've listed everything from the MIRI research guide that I could perceive as a distinct category.

(ex. If you were learning about Digital Systems and Computer Architecture, my test would be "In systemVerilog, simulate a basic 16-bit processor that can be programmed using a RISC assembly language of your design.")

_(edit: I know that "huge swaths" is pretty vague, and suggestions don't have to be things that you think certify/prove that you get a topic. While a "comprehensive test" would be nice, problems/prompts like what Qiaochu commented are exactly what I'm looking for)_

  

1.  Set theory
2.  Probability
3.  Probabilistic Inference
4.  Statistics
5.  Machine Learning
6.  Solomonoff Induction
7.  Naturalized Induction
8.  VNM Decision Theory
9.  Functional Decision Theory
10.  Logical Uncertainty
11.  First Order Logic
12.  Vingean Reflection
13.  Corrigibility
14.  Linear Algebra
15.  Topology
16.  Category Theory
17.  Type Theory
# Expertise Exchange

There are multiple fields of knowledge that are very hard to learn about and where it's hard to find a rational person to give you good answers about the field of knowledge. 

I want to start this thread to give people the opportunity to request expertise in certain domains that they want to know more about but have a hard time finding good sources on.
# Using lying to detect human values

In my current research, I've often re-discovering things that are trivial and obvious, but that suddenly become mysterious. For instance, it's blindingly obvious that the [anchoring bias](http://facweb.plattsburgh.edu/wendy.braje/students/psy205/JKarticle.pdf) is a bias, and almost everyone agrees on this. But this becomes puzzling when we realise that there is [no principled ways](https://www.lesserwrong.com/posts/rtphbZbMHTLCepd6d/humans-can-be-assigned-any-values-whatsoever) of [deducing the rationality and reward](https://arxiv.org/abs/1712.05812) of irrational agents.

Here's another puzzle. Have you ever seen someone try and claim that they have certain values that they manifestly don't have? Seen their facial expression, their grimaces, their hesitation, and so on.

There's an immediate and trivial explanation: they're lying, and they're doing it badly (which is why we can actually detect the lying). But remember that there is no way of detecting the preferences of an irrational agent. How can someone lie about something that is essentially non-existent, their values? Even if someone knew their own values, why would the tell-tale signs of lying surface, since there's no way that anyone else could ever check their values, even in principle?

But here evolution is helping us. Humans have a self-model of their own values; indeed, this is what we use to [define what those values are](https://www.lesserwrong.com/posts/kmLP3bTnBhc22DnqY/beyond-algorithmic-equivalence-self-modelling). And evolution, being lazy, re-uses the self-model to interpret others. Since these self-models are broadly similar from person to person, people tend to agree about the rationality and values of other humans.

So, because of these self-models, our own values "feel" like facts. And because evolution is lazy, lying and telling the truth about our own values triggers the same responses as lying or telling the truth about facts.

This suggests another way of accessing the self-model of human values: train an AI to detect human lying and misdirection on factual matters, then feed that AI a whole corpus of human moral/value/preference statements. Given the [normative assumption](https://www.lesserwrong.com/posts/Fg83cD3M7dSpSaNFg/normative-assumptions-regret) that lying on facts resembles lying on values, this is another avenue by which AIs can learn human values.

So far, I've been assuming that human values are a single, definite object. In my next post, I'll look at the messy reality of under-defined and contradictory human values.
# AI Summer Fellows Program

CFAR and MIRI are running a free AI Summer Fellows Program (AISFP) in the San Francisco Bay Area from June 27 to July 14. Aimed at increasing participants' abilities to do technical research in AI alignment, the program includes CFAR's applied rationality content as well as practice in doing technical research toward AI Safety with MIRI researchers and (20-24) other participants.

**Program Description**

AISFP, a two week summer program designed to increase participants' ability to do technical research into [the AI alignment problem](https://futureoflife.org/ai-open-letter), will take place in the San Francisco Bay Area from June 27 to July 14.

The intent of the program is to boost participants, as far as possible, in four skills:

1\. The CFAR applied rationality skillset, including both what is taught at our intro workshops, and more advanced material from our alumni workshops.

2\. Epistemic rationality as applied to the foundations of AI, and other philosophically tricky problems — i.e., the skillset taught in [the core LW Sequences](https://wiki.lesswrong.com/wiki/Sequences#Core_Sequences) (e.g. reductionism and how to reason in contexts as confusing as anthropics without getting lost in words).

3\. Technical forecasting in AI as well as AI alignment interventions (e.g. the content discussed in Nick Bostrom’s book _[Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/ref=dp_kinw_strp_1)_).

4\. The ability to do AI alignment-relevant technical research, while reflecting on the cognitive habits involved. We will give crash courses in: reflection, logical uncertainty, and decision theory.

Finalists will be contacted by a MIRI staff member for an interview.

\[6/8: Applications to AISFP18 now closed\]
# You can now log in with your LW1 credentials on LW2

I just finished an update that ported over the old passwords from LW1 while still preserving security (see my previous meta post). We only ported passwords over from sometime in last May, and will run the same import on the new database if the vote completes and we are doing the final transfer, but I figured I would let people know now and tell me of any problems they encounter in the meantime.

The old passwords were hashed with a relatively weak hashing function (SHA1), so I do recommend that people change their passwords after they login with their old accounts. There is also some risk of someone impersonating you if they got access to the old LW1 database (since with the new setup the old password hash is sufficient to allow them to log in), so if you change passwords this problem is also resolved.
# Deciphering China's AI Dream

> This report examines the intersection of two subjects, China and artificial intelligence, both of which are already difficult enough to comprehend on their own. It provides context for China’s AI strategy with respect to past science and technology plans, and it also connects the consistent and new features of China’s AI approach to the drivers of AI development (e.g. hardware, data, and talented scientists). In addition, it benchmarks China’s current AI capabilities by developing a novel index to measure any country’s AI potential and highlights the potential implications of China’s AI dream for issues of AI safety, national security, economic development, and social governance.

> The author, Jeffrey Ding, writes, “The hope is that this report can serve as a foundational document for further policy discussion and research on the topic of China’s approach to AI.” The report draws from the author’s translations of Chinese texts on AI policy, a compilation of metrics on China’s AI capabilities compared to other countries, and conversations with those who have consulted with Chinese companies and institutions involved in shaping the AI scene.
# AI Summer Fellows Program

CFAR and MIRI are running a free AI Summer Fellows Program (AISFP) in the San Francisco Bay Area from June 27 to July 14. Aimed at increasing participants' abilities to do technical research in AI alignment, the program includes CFAR's applied rationality content and practice in doing technical research toward AI Safety with MIRI researchers and (20-24) other participants.

**Program Description**

AISFP, a two week summer program designed to increase participants' ability to do technical research into the [AI alignment problem](http://futureoflife.org/misc/open_letter), will take place in the San Francisco Bay Area from June 27 to July 14.

The intent of the program is to boost participants, as far as possible, in four skills:

1.  The CFAR applied rationality skillset, including both what is taught at our intro workshops, and more advanced material from our alumni workshops.
2.  Epistemic rationality as applied to the foundations of AI, and other philosophically tricky problems — i.e., the skillset taught in the [core LW Sequences](https://wiki.lesswrong.com/wiki/Sequences#Core_Sequences) (E.g.: reductionism; how to reason in contexts as confusing as anthropics without getting lost in words).
3.  Technical forecasting in AI as well as AI alignment interventions (e.g., the content discussed in Nick Bostrom’s book [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/ref=dp_kinw_strp_1)).
4.  The ability to do AI alignment-relevant technical research, while reflecting on the cognitive habits involved. We will give crash courses in: reflection, logical uncertainty, and decision theory.

Finalists will be contacted by a MIRI staff member for an interview.
# LW Migration Announcement

The votes are in, and of the eligible 376 voters, 102 voted to Migrate and 15 voted to archive. With 87% in favor, we’re going ahead with the transition, which will begin tomorrow (3/22) at 6pm Pacific time. Trike will take a snapshot of the database, transfer it to us, and we’ll begin the database import of new material since the last import. At 7pm, the DNS server will point to our server instead of Trike’s, and so people visiting lesswrong.com will see the new site. The import will run in parallel with the site operating, so there won’t be any downtime of the new site.

Some things to note:

1.  Old links will continue to work, redirecting to the right new URL. We’ll be watching the analytics to notice any failures, and you can also alert us through Intercom (which we’ll shut off a few days into the transition, as problems get fixed).
2.  Lesserwrong links will redirect to the same page at lesswrong.
3.  We’ll add old karma scores to new karma scores, and may adjust the vote weight algorithm accordingly.
4.  Please report issues [here](https://github.com/Discordius/Lesswrong2/issues).

Also, for those of you near Berkeley, we’ll be throwing a launch party on April 7th at 7pm ([LW](https://www.lesserwrong.com/posts/udJ3bnm5bp9aQKaS5/lesswrong-launch-party), [Facebook](https://www.facebook.com/events/510652962670039/)).
# LessWrong.com URL transfer complete, data import will run for the next few hours

As you can easily see when you visit LessWrong.com, the site transfer is complete, and we are now on the new URL. The data import will run for the next few hours, and by tomorrow morning everything should be complete.

Let us know if you run into any problems, find any broken links, or anything of that sort!
# Systemizing and Hacking

Let's call two broad clusters of problem-solving "systemizing" and "hacking." Systemizing roughly refers to building general, robust solutions. Hacking refers to quickly building one-offs. You would expect to accrue technical debt while hacking and pay it down while systemizing.

Both are useful in different situations. A big win for systemizing would be finding patterns in code and dramatically reducing the complexity by factoring out those patterns. A big win for hacking would be quickly testing several UIs to find one that potential customers love.

As you can probably guess, I lean very heavily towards systemizing, and have typically sucked at hacking – I generally relied on coworkers for that skill. But I started working on solo projects where I couldn't easily rely on someone else, so naturally, I developed a system for hacking things together.

The concrete example here is that I wanted to be able to run thousands of completely independent machine learning jobs quickly in parallel across a lot of computers, and I was looking into AWS Batch to do this.

Here's how I would have approached this in "systemizing" mode:

1.  Break down all the parts involved, e.g. AWS Batch, Docker, boto3, etc.
2.  Study them in detail until I grok each one, writing small bits of code primarily designed to help me understand
3.  Assemble a solution

Here's how I did this in hacking mode:

1.  Find a valuable point of high uncertainty (e.g., running Python scripts through Docker)
2.  Find something that does something similar to what I want (e.g., an online introduction to Docker that involved running a particular Python script)
3.  Modify that until it does what I want (successfully running my Python script and uploading data to AWS)
4.  Find the next valuable point of uncertainty and repeat (running a single Docker image on AWS Batch)

This was really effective – previously, I might have spent ~40 hours deeply learning all the tools involved before I got the system up and running. Using the hacking approach, it only took me about 3 hours – literally an order of magnitude faster.

Of course, the hacking approach has clear disadvantages – I don't deeply understand Docker or any of the tools involved, and my experience here won't save me a ton of time if I work on a similar project. But in this case, I don't expect to do anything similar in the near-term, so hacking was a huge win. More generally, having the skill of hacking things together gives me many more options for projects in the future.
# Idea: Open Access AI Safety Journal

This is short because I'm mainly trying to gather feedback on how much the idea might be worth pursuing, but how much value does it seem the existence of an open access AI safety journal would provide? Some reasons I can think of in favor:

*   Have a journal focused on AI safety, giving the field better visibility.
*   Peer review from researchers active in AI safety research.
*   Open access venue for AI safety research to avoid ethical concerns with publishing in closed journals.
*   A venue where AI safety ideas too far outside the mainstream but are academically rigorous can be published.
*   Give AI safety researchers a respectable venue for publishing to satisfy their career needs while reducing the effort they have to expend to get their work published, allowing more AI safety research to happen.

Some reasons against:

*   There are already plenty of journals.
*   Pre-prints are good enough.
*   Journals are a waste of time to signal quality that is generally easy to assess from reading articles themselves.
*   Most of the value could be had by regularly publishing a reviewed list of pre-prints and otherwise published articles in AI safety.
*   Drag on time of folks who could be doing research instead of reviewing and editing articles.

It appears that starting an open access journal is a little tedious but not especially hard, sc. this [step-by-step guide](https://www.martineve.com/2012/07/10/starting-an-open-access-journal-a-step-by-step-guide-part-1/), so it could probably be done by a team of 1-3 volunteers, but hopefully could be professionalized if it takes off.

Thoughts?
# The most important step

I have a penchant for the stories of the old, in the era before the Bayesian Enlightment and before the Singers and Cantors surfaced, the days when the students of the Art of Rationality met underground, away from prying eyes.

Without the Sacred Texts to guide them, the disciples of Rationality spent long times discussing and refining the Art. 'Twas an arduous task, yet they persevered, guided by the promise of Light and Truth, the glimpse of Polaris that decided their course.

In this story in particular, two disciples of the Temple of the Bay discussed about the most important step in a quest.

-Surely - said one of them - the most important step must be first; for no quest has come to fruition without going through the first step.

-But you forget - said the other - the same reasoning applies to the last. No quest is complete without its last step.

-What about the value of the journey? It is infinitely more valuable to get halfway through a sacred quest, and then fail the last step, than to never have embarked upon it to begin with.

-In the Art, all that matters is victory. We do not embark in the path of Rationality to try, but to reach that which really matters. One does not get points for effort. Only conclusions matter.

-How can you even talk about the last step? In the quests that really matter, we should not see the end. Those quests, like the pursue of the Art, are powerful, alive things. To seek to end them is to mean their death.

Both kept arguing about both things for ten days and ten nights, until the master of the Temple came back from a journey of sacred meditation, to find both her disciples in heated conversation.

-Silence! - said the master - You both have brought dishonor to this sacred house! You have unattended your tasks while talking about abstract things, and thus you have lost sight of the Enemy.

-But master - protested one disciple - this is important theory to be crafted! Our quests must wait, for as you taught us, a strike without purpose is a wasted motion.

-Had you really understood the lesson - said the master - you would clearly see how to [dissolve this silly question](https://www.lesswrong.com/posts/Mc6QcrsbH5NRXbCRX/dissolving-the-question). No step, nor the first one nor the last, is as important as the next one!

And the students were Enlightened.
# Diffusing "I can't be that stupid"

Why does it hurt so much when I think that people I care about are very wrong about something I think is important? If I'm talking with someone I'm not super close with, I'm usually very capable of not getting emotional or riled up when we find important disagreements.

After some though, it seems like my mind is running a process similar to the one below.

![](https://i.imgur.com/YTNafrF.png)

Quick summary:

For some reason, I don't want to accept that someone can matter and be worthy of care, and still be (from what I know) very wrong about very important things. In order to work around this, I slightly dehumanize people when I think they're majorly wrong. When I think someone I care about is majorly wrong, I bounce around in a loop

I'd guess that the experience of pain and discomfort that comes from disagreeing with close ones about important things is produced by spending time stuck in the loop on the left.

I should note that it feels uncomfortable to say that the above process is one that I routinely use, primarily because of how obviously broken it is upon reflection. You'd never catch me audibly saying, or even explicitly thinking, "Dante doesn't matter/isn't human because he doesn't agree with me."

There was a non-trivial delay between when I first though I could be doing some sort of dehumanization, and when I seriously considered it as a possibility. I don't like how long that took. I think bugs like this can often be hard to deal with because there's a sense of, "_There's no way I could be that stupid."_

Sometimes it seems like the mechanism is, "I would feel bad if I turned out to be 'that stupid', and I don't want to feel bad, so I'm not going to consider it." But it seems like there's also plenty of instances where it's, "I genuinely assign a low probability to being 'that stupid', and so I'm not going to investigate it."

When I queried this belief, I found something like, "Given that I am capable of feats of great intelligence, all parts of me should be very intelligent, because that's how intelligent things work\[citation not need because I'm an implicit belief\]." Which again, when I say it out loud I can see right through that.

Hmmmm... wait, so is my conclusion just that you should pay attention to your beliefs, and then ask, "[Why do you believe what you believe?](https://wiki.lesswrong.com/wiki/Fundamental_Question_of_Rationality)"

It sure is :)

Process to Apply This Post

*   Work on noticing the feeling of, "I can't be that stupid."
*   Fully describe the behavior that you thing you "aren't stupid" enough to be doing.
*   Grill the validity of the belief that you "aren't that stupid".
*   Work on correcting the behavior.
*   Profit

Feel free to comment with examples from your own experience.
# GreaterWrong—several new features & enhancements

_(I hope it’s acceptable to post this in the Meta section; if not, the mods can move it to my personal blog.)_

[GreaterWrong.com](https://www.greaterwrong.com/) (which is an [alternative way to browse the new LessWrong](https://www.lesserwrong.com/posts/66DXhQJyPEJNsXgfw/an-alternative-way-to-browse-lesswrong-2-0)) has been adding some new features and enhancements—here’s some of the stuff we’ve got now:

*   Appearance customization (click the “sliders” button on the top-left)
    *   Images in posts are now properly excluded from changes to brightness / contrast / inversion / etc.
*   Adjustable text size (look on the top right, just under the content width selector buttons)
    *   _Note:_ This doesn’t work in Firefox, unfortunately.
*   Most functions / buttons / etc. now have [accesskeys](https://en.wikipedia.org/wiki/Access_key), letting you do a lot of navigation (and other things) via the keyboard
    *   All available accesskeys are [listed on the About page](http://www.obormot.net:4242/about#keyboard-based-navigation)
*   Pagination links at the top of all listings pages (next page, previous page, etc.)
*   Improved mobile layout (especially for the post/comment editing UI)
*   An [updated About page](https://www.greaterwrong.com/about) with detailed info on many of GreaterWrong’s features

As always, please [let us know](https://www.greaterwrong.com/about) if you’re having any problems with the site, bugs, etc. (If you do report bugs or problems, please include your browser, version, and operating system!)
# The master skill of matching map and territory

Robin Hanson tells us to have fewer opinions on things, to specialize and be agnostic about everything outside your field of expertise. This may be good advice, but most of us won't take it. We're too obsessed with being right about stuff, including things we didn't study. Whether or not there is an external reason why being correct about something is useful, we usually care either way.

Most of our beliefs are based on what other people say, so the key skill seems obvious: identifying which people's views to consider strong evidence. Mastering that means being on par with the greatest experts in every field – not in understanding of the field itself, but in accuracy of views on controversial topics.

Why is there so little talk about this? Maybe because it's controversial, or it could be a status thing. I know I'd be uncomfortable with one entry if I had to share a short list of people who have a sizeable effect on my world-view. But even if that's enough of a reason to avoid talking about conclusions, about which particular people are or aren't trustworthy, it shouldn't stop us from at least going meta.

So the second half of the post will be my first suggestion, a list of cues which I, on reflection, appear to follow, to determine whether or not to take another person's views seriously. I'm almost certainly missing important ones, most likely even some that I use myself but am not conscious of. Items are phrased as actions: the person to be evaluated does X. Cues 1-8 are positive and improve trustworthiness, cues 9-12 are negative – though in reality most of them represent a spectrum and could also be phrased in the opposite way.

  

* * *

1\. Is internally consistent

2\. Is aligned with things I'm already confident in

3\. Brings up points that aren't obvious but make sense

4\. Has high IQ

5\. Uses implicitly or explicitly consequentialist arguments

6\. Uses sentences that in isolation sound as sophisticated as necessary to make the point and not more

7\. Performs an unusually low amount of visible signaling

8\. Cites people I consider to be highly trustworthy

* * *

9\. Assumes a high baseline of trustworthiness of other people based on their social status or academic degrees

10\. Ascribes well-thought-out views to groups of people that haven't been selected through a plausible filter

11\. Bases her judgment of other people largely on the degree of agreement between her and their views.

12\. Ascribes malicious intent to groups of people that haven't been selected through a plausible filter

And of course there is a prior of trustworthiness based on past actions which may dominate any other cue.
# The fundamental complementarity of consciousness and work

Matter can experience things. For instance, when it is a person. Matter can also do work, and thereby provide value to the matter that can experience things. For instance, when it is a machine. Or also, when it is a person.

An important question for what the future looks like, is whether it is more efficient to carry out these functions separately or together.

If separately, then perhaps it is best that we end up with a huge pile of unconscious machinery, doing all the work to support and please a separate collection of matter specializing in being pleased.

If together, then we probably end up with the value being had by the entities doing the work.

I think we see people assuming that it is more efficient to separate the activities of producing and consuming value. For instance, that the entities whose experiences matter in the future will ideally live a life of leisure. And that lab grown meat is a better goal than humane farming.

Which seems plausible. It is at least in line with the general observation that more efficient systems seem to be specialized.

However I think this isn’t obvious. Some reasons we might expect working and benefiting from work to be done by overlapping systems:

*   We don’t know which systems are conscious. It might be that highly efficient work systems tend to be unavoidably conscious. In which case, making their experience good rather than bad could be a relatively cheap way to improve the overall value of the world.
*   For humans, doing purposeful activities is satisfying, so much so that there are concerns about how humans will cope when they are replaced by machines. It might be hard for humans to avoid being replaced, since they are probably much less efficient than other possible machines. But if doing useful things tends to be gratifying for creatures—or for the kinds of creatures we decide are good to have—then it is less obvious that highly efficient creatures won’t be better off doing work themselves, rather than being separate from it.
*   Consciousness is presumably cheap and useful for getting something done, since we evolved to have it.
*   Efficient production doesn’t seem to evolve to be entirely specialized, especially if we take an abstract view of ‘production’. For instance, it is helpful to produce the experience of being a sports star alongside the joy of going to sports games.
*   Specialization seems especially helpful if keeping track of things is expensive. However technology will make that cheaper, so perhaps the world will tend less toward specialization than it currently seems. For instance, you would prefer plant an entire field of one vegetable than a mixture, because then when you harvest them, you can do it quickly without sorting them. But if sorting them is basically immediate and free, you might prefer to plant the mixture. For instance, if they take different nutrients from the soil, or if one wards of insects that would eat the other.
# Charting Deaths: Reality vs Reported

Some of you may recall an iconic study Yudkowsky references in his [Availability](https://www.lesswrong.com/posts/R8cpqD3NA4rZxRdQ4/availability) essay, where Combs and Slovic looked at the different ways people died and newspaper reporting to see if they were proportional.

For my data science course at UCSD, my group and I attempted to replicate those findings with some more recent data!

(The link has lots more graphs, analysis, and links to the code used.)

![](https://i.redd.it/5sq16d2moso01.gif)
# Rationalist Lent is over

[Rationalist Lent is over](https://www.lesswrong.com/posts/KZ47RTGvgpH3PhiFR/rationalist-lent). How did it go for everyone? Did you learn anything about yourself / superstimuli / pica / etc.?

I had a pretty good time giving up porn, which I stuck to. I also masturbated much more sparingly, which had [interesting and pleasant effects on my sex drive](https://www.nateliason.com/testosterone-timing/). This is part of a longer project I'm engaging in of getting more in touch with my sexuality and it's been going well so far. I have an increasingly strong sense now of my past porn consumption as having been pretty bad for me - in machine learning terms, it's [adversarial training data](https://blog.openai.com/adversarial-example-research/), very closely analogous to junk food. Porn, like many of my previous addictions (mostly various forms of fiction), has also felt much less tempting over the last 2 months because my life has gotten a lot better. Long story.
# Reward hacking and Goodhart’s law by evolutionary algorithms

Nice collection of anecdotes from the Evolutionary Computation and Artificial Life research communities about evolutionary algorithms subverting researchers intentions, exposing unrecognized bugs in their code, producing unexpected adaptations, or exhibiting outcomes uncannily convergent with ones in nature. Some of my favorites:

In other experiments, the fitness function rewarded minimizing the difference between what the program generated and the ideal target output, which was stored in text files. After several generations of evolution, suddenly and strangely, many perfectly fit solutions appeared, seemingly out of nowhere. Upon manual inspection, these highly fit programs still were clearly broken. It turned out that one of the individuals had deleted all of the target files when it was run! With these files missing, because of how the test function was written, it awarded perfect fitness scores to the rogue candidate and to all of its peers

...

To test a distributed computation platform called EC-star \[84\], Babak Hodjat implemented a multiplexer problem \[85\], wherein the objective is to learn how to selectively forward an input signal. Interestingly, the system had evolved solutions that involved too few rules to correctly perform the task. Thinking that evolution had discovered an exploit, the impossibly small solution was tested over all possible cases. The experimenters expected this test to reveal a bug in fitness calculation. Surprisingly, all cases were validated perfectly, leaving the experimenters confused. Carefully examination of the code provided the solution: The system had exploited the logic engine’s rule evaluation order to come up with a compressed solution. In other words, evolution opportunistically offloaded some of its work into those implicit conditions.
# Harry Potter and the Method of Entropy

I did some [Aversion Factoring](https://radimentary.wordpress.com/2018/02/04/hammertime-day-7-aversion-factoring/) and found that my main aversion to writing math is the terrible LaTeX support on WordPress. For now, I’ll try writing things up offline and posting PDFs.

HPMoE is a sequence on the entropy method in combinatorics. It should be fun to think about for anyone interested in information theory and probability as well.

[HPMoE1](https://radimentary.files.wordpress.com/2018/03/hpmoe1.pdf "HPMOE1")

Please post solutions to the exercises in ROT13.
# Reframing misaligned AGI's: well-intentioned non-neurotypical assistants

I think when people imagine misaligned AGI's, they tend to imagine a superintelligent agent optimizing for something other than human values (e.g. paperclips, or a generic reward signal), and mentally picture them as adversarial or malevolent. I think this visualization isn't as applicable for AGI's trained to optimize for human approval, like [act-based agents](https://ai-alignment.com/act-based-agents-8ec926c79e9c), and I'd like to present one that is.

If you've ever employed someone or had a personal assistant, you might know that the following things are consistent:

*   The employee or assistant is genuinely trying their hardest to optimize for your values. They're trying to understand what you want as much as they can, asking you for help when things are unclear, not taking action until they feel like their understanding is adequate, etc.
*   They follow your instructions literally, under a sensible-to-them-seeming interpretation completely different from your own, and screw up the task entirely.

Suppose you were considering hiring a personal assistant, and you knew a few things about it:

*   Your assistant was raised in a culture completely different from your own.
*   Your assistant is extremely non-neurotypical. It doesn't have an innate sense of pain or empathy or love, it's a savant at abstract reasoning, and it learned everything it knows about the world (including human values) from Wikipedia.
*   Your assistant is in a position where it has access to _enormous_ amounts of resources, and could easily fool you or overpower you if it decided to.

You might consider hiring this assistant and trying really, really hard to communicate to it exactly what you want. It seems like a way better idea to just _not hire_ this assistant. Actually, you’d probably want to run for the hills if you were forced to. Some specific failure modes you might envision:

*   Your assistant's understanding of your values will be weird and off, perhaps in ways that are hard to communicate or even pin down.
*   Your assistant might reason in a way that looks convoluted and obviously wrong to you, while looking natural and obviously correct to it, leading it to happily take actions you'd consider catastrophic.

As an illustration of the above, imagine giving an eager, brilliant, extremely non-neurotypical friend free rein to help you find a romantic partner (e.g. helping you write your OKCupid profile and setting you up on dates). As another illustration, imagine telling an entrepreneur friend that superintelligences can kill us all, and then watching him take drastic actions that clearly indicate he's missing important nuances, all while he misunderstands and dismisses concerns you raise to him. Now reimagine these scenarios with your friends _drastically_ more powerful than you.

_This is my picture of what happens by default if we construct a recursively self-improving superintelligence by having it learn from human approval. The superintelligence would not be malevolent the way a paperclip maximizer would be, but for all intents and purposes might be._
# LW Update 3/31 - Post Highlights and Bug Fixes

In the last few weeks we implemented several bug fixes, including:

*   Login form gives more useful error messages
*   Fixed most broken post links from the Lesswrong migration. (There are still some issues surrounding linked comments, and old images)
*   Notifications don't appear twice if you reply to your own comment or someone replies to a comment of yours on a post that you created
*   Sequence navigation links at the bottom of the sequence post now properly keep you "in the sequence" (before, they'd send you to the next post, but then the next post wouldn't be in the sequence, just a standalone post). \[There are still occasional issues with sequence links at the _end_ of sequence, where the behavior is a bit more complicated because sometimes you're supposed to go to the _next_ sequence and sometimes you're not\]
*   Some backend fixes to improve the search functionality.

Usernames
---------

Usernames on the frontpage are clickable again. (We disabled this awhile ago because it was leading to a lot of misclicks. We've changed the Post List Items quite a bit since then, and had heard some complaints about not being able to click on usernames to lead to a user's profile. We'll see how it goes this time)

Highlights
----------

Whenever you publish or edit a post, it now generates a highlight based on the first 2000 characters. In the Recent Discussion section, posts that you haven't read yet will display the highlight so you can get some context before reading comments (the post will be marked with a green dot if you haven't read the post yet, which will disappear as soon as you've expanded the highlight once)

New posts will appear in the recent discussion section (to reduce a weird incentive to comment on your own post to get it onto the frontpage if it was a meta or personal blogpost. The recent discussion section will essentially serve as a chronological list of which posts most recently had some kind of activity)
# My take on agent foundations: formalizing metaphilosophical competence

I have some rough intuitions about the purpose of MIRI's agent foundations agenda, and I'd like to share them here. (Note: I have not discussed these with MIRI, and these should not be taken to be representative of MIRI's views.)

I think there's a common misconception that the goal of agent foundations is to try building an AGI architected with a decision theory module, a logical induction module, etc. In my mind, this is completely not the point, and my intuitions say that approach is doomed to fail.

I interpret agent foundations as being more about providing formal specifications of [metaphilosophical competence](https://www.lesserwrong.com/posts/CCgvJHpbvc7Lm8ZS8/metaphilosophical-competence-can-t-be-disentangled-from), to:

*   directly extend our understanding of metaphilosophy, by _adding conceptual clarity_ to important notions we only have fuzzy understandings of. (Will this agent fall into epistemic pits? Are its actions low-impact? Will it avoid catastrophes?) As an analogy, formally defining mathematical proofs constituted significant progress in our understanding of mathematical logic and mathematical philosophy.
*   allow us to _formally verify_ whether a computational process will satisfy desirable metaphilosophical properties, like those mentioned in the above parenthetical. (It seems perfectly fine for these processes to be built out of illegible components, like deep neural nets—while that makes them harder to inspect, it doesn't preclude us from making useful formal statements about them. For example, in [ALBA](https://ai-alignment.com/alba-an-explicit-proposal-for-aligned-ai-17a55f60bbcf), it would help us make formal guarantees that distilled agents remain aligned.)

I want to explore [logical induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/) as a case study. I think the important part about logical induction is the logical induction _criterion_, _not_ the algorithm implementing it. I've heard the implementation criticized for being computationally intractable, but I see its primary purpose as showing the logical induction criterion to be _satisfiable at all._ This elevates the logical induction criterion over all the other loose collections of desiderata that may or may not be satisfiable, and may or may not capture what we mean by logical uncertainty. If we were to build an actual aligned AGI, I would expect its reasoning process to satisfy the logical induction criterion, but not look very much like the algorithm presented in the logical induction paper.

I also think the logical induction criterion provides an exact formalization—a necessary AND sufficient condition—of what it means to not get stuck in any epistemic pits in the limit. (The gist of this intuition: epistemic pits you're stuck in forever correspond exactly to patterns in the market that a trader could exploit forever, and make unbounded profits from.) This lets us formalize the question "Does X reasoning process avoid permanently falling into epistemic pits?" into "Does X reasoning process satisfy the logical induction criterion?"