Harley-ml/lesswrong huggingface dataset:
# Offloading Executive Functioning to Morality

tl;dr- "My executive functioning doesn't work, so I use morality instead."  
  
From an outside perspective it's a little hard to tell that I don't have a lot of executive functioning. I keep my living space relatively clean, I get places relatively on time, I maintain a regular 9-to-5, and I have a lot of side projects in my free time that I generally manage to complete. But my life wasn't always that way, and most of the general rationality/CFAR methods of coping don't work at all for me. Getting diagnosed as an adult with ADD and trying a variety of medications also didn't work at all for me. Obviously, generalizing from one mind is a particularly Bad Idea in my case as I'm rather an outlier, but hopefully things that work for me might be useful for other people. (This may end up as a sequence of Things That Work for Me, but for today I'm just focusing on one.)  
  
One way I manage to function is to have employment or other life aspects where I feel it is a Moral Imperative to Do The Thing. I call this "Offloading my executive functioning onto my morality," and it pretty much means setting up my life in a way where Doing Things comes naturally, because otherwise there are immediate negative effects on people I care about. I don't want to get into EvPsych stuff, but I feel it is a more natural motivation and How Humans Are Wired.  
  
**A bunch of examples:**  
  
**Bad Idea:** Have a job that lets you arrive whenever. Your arrival time has no effect on anyone else's work.  
  
**Good Idea:** Have a job where a specific person is depending on you to arrive on time. Maybe your friend can't leave his shift until you arrive for yours. Maybe you are teaching a karate class and arriving late is wasting a bunch of people's time.  
  
\-\-\-\-\-\-\-\-\-\-  
**Bad Idea:** RSVP to events with "Maybe", so that no one's put out if you don't show.  
  
**Good Idea:** RSVP to events with "Yes, AND I need a ride" or "Yes, AND I'll bring dessert" so that not showing up is causing a detriment to other people.  
  
\-\-\-\-\-\-\-\-\-\-\-\-  
**Bad Idea:** Have a job like academia where even if your research might be hugely useful, it is theoretic in application and far in the future.  
  
**Good Idea:** Have a job like teaching or nursing, where people are relying on you, and not-working has immediate, visible, detrimental results for other people.

\-\-\-\-\-\-\-\-\-\-\-\-\-

In a more traditionally rational framing, these can be summarized as: Use social pre-commitment techniques, and keep things in near mode (Far mode morality does not work for this use for me). But the framing of "Use Morality Instead" tends to point more naturally at the thing that works for me.  
  
Note: A possible downside of this is that I am much less tolerant of things like "People showing up really late for a one-on-one meeting", because my method of functioning assigns those behaviors as "Morally Bad".
# Why no total winner?

Why doesn't a single power rule the world today?

\[I'm taking advantage of the new "LW posts as blog posts" format to post something I'm pretty unsure about. I'm working from my memories of the blog posts, and from a discussion I had with Robin Hanson and Katja Grace in late 2012. Please let me know if any of this is inaccurate!\]

One of the key differences of opinion in the [Hanson-Yudkowsky AI-Foom Debate](https://intelligence.org/ai-foom-debate/) is about the idea of a "decisive advantage". If I'm not misrepresenting the parties horribly, the idea is that some point in a world with AGI, some AGI-enabled party uses their greater intelligence to increase their general power: money, resources, control over others, intelligence and suchlike. That greater power increases their ability to gain power, resulting in a snowball effect that ends with some party having control over the outcomes for all of Earth-originating life.

Robin asks the very reasonable question: if that's how things work, why hasn't it already happened? What stops the largest business using its decisive power over smaller ones to defeat and absorb them, growing ever larger and more powerful until all other businesses fall to its power? Why do we have multiple nations today, when this model would seem to predict that a single state should ultimately conquer and rule all? I don't remember Robin proposing an answer of his own: a mechanism or theoretical model that would lead us to expect multiple powers. But it seems like a good question, and it's bugged me ever since.

I think I'd need to be much more of a student of history than I am to have any confidence in an answer, so let me share some wild speculation that might at least start discussion:

*   Regulation: Such growth isn't an option for legal businesses at all, because states exist. So powerful a business would challenge the power of the state, and the state is in a position to disallow that. The explicit purpose of monopoly legislation is to stop a business which has become very powerful in one area from leveraging that to become powerful elsewhere.
    
*   Principal-agent problems: it sure would be easier to keep an empire together if you could reliably appoint generals and rulers who always did what you told them to. Especially if the round-trip time for getting them a message is on the order of weeks, and you have to entrust them with the discretion to wield tremendous power in the mean time.
    
*   Moral norms: Nuclear weapons gave the USA a decisive advantage at the end of WWII. If the USA had been entirely ruthless and bent on power at any cost, it would immediately have used that advantage to cripple all rivals for world superpower and declared its rulership of the world.
    

I don't expect any of these factors to limit the growth of an AGI. Is there some more general limit to power begetting power that would also affect AGI?
# Multidimensional signaling

What do you infer about a person who has ugly clothing? Probably that they have poor taste  (in clothes, or subcultures). But it could also be that they are too poor to improve their wardrobe. Or can’t be bothered.

What about someone with poor grades? The obvious inference is that they aren’t so capable at the subject, but it may again be that they can’t be bothered, or that they have more urgent things to do with their time.

And someone who makes clever jokes? Probably that they are smart and naturally funny,  but if they had more time or effort to spend on this, it probably helped.

For all kinds of traits that people might try to signal with their behavior, someone can send a better signal if they have more money or time or self-control. Even when the main signal being sent is not usually thought to be about any of those things.

The reason this interests me: if signals often divide the population into ‘better or richer’ vs. ‘worse or poorer’, I wonder if this would cause us to imagine that being rich is associated with being better, even if the two were entirely independent. (And similarly for wealth in other general-use resources, like self-control and time).

In a simple case, suppose there are just people with pretty clothes (who are both rich and have good taste) and people with ugly clothes (who either have bad taste, or lack resources or will). Then do observers come to think of ‘rich good taste’ type and a ‘poor bad taste’ type? Or do they pay more attention to the actual structure of the space, and know for instance that this doesn’t mean learning that someone really has bad taste actually means they are probably poorer.

![New Doc 2017-10-15 (1)](https://meteuphoric.files.wordpress.com/2017/10/new-doc-2017-10-15-1.jpg?w=500)

Note that I’m not merely suggesting that a person with more wealth can send signals to look like they are better—that much is clear. I’m suggesting that at a population level, if the wealthier people can’t be distinguished from the better people on some axis, then observers may come to think that the two are associated in general, even if they are not at all.

If so, this would be important, because it would apply in a huge range of cases of signaling. So that the properties of poverty and weak-willedness and such would appear to us to be much worse than they really were.
# Logical Updatelessness as a Robust Delegation Problem

(Cross-posted on [AgentFoundations.org](https://agentfoundations.org/item?id=1689))

There is a cluster of problems in understanding naturalized agency which I call Robust Delegation. It refers to the way in which a limited agent might direct a more powerful agent to do its bidding. The limit agent starts out in control because it gets to design the more powerful agent, but it is not easy to retain that power. In most applications the limited agent is a human and the powerful agent is an AI.

Example of problems in this subfield include corrigibility, value learning, and informed oversight.

Logical Updatelessness is one of the central open problems in decision theory. An updateless agent is one that does not update on its observations, but instead chooses what action it wants itself to output upon being input those observations. It can use this to get counterfactually mugged (which is a good thing). This is basically the only way we know how to make a reflectively stable agent.

The problem is that we do not know how to translate updatelessness into a logical setting. In the toy models in which we can define updatelessness, the agent is logically omniscient and just makes empirical observations. Thus, starts out just as powerful as it will ever be. In fact, we can view it as though the only agent is the agent existing at the beginning of time. That agent chooses a policy, a function from inputs to outputs, and all the future agents can just blindly follow that policy without thinking.

In the logical setting, the future agents are smarter than the past agents. They learn more logic in addition to making empirical observations. The agent at the beginning time does not have the power to contain a policy from all the logical observations to actions. This agent does not choose a simple policy to follow in the future, but instead chooses a complicated program to run in the future.

I claim that logical updatelessness is hard while empirical updatelessness is easy because an empirically updateless agent is deferring control to a simple policy that it understands, while a logically updateless agent has to defer control to a more powerful agent that it does not understand. Thus, logical updatelessness can be viewed as a robust delegation problem.

If we think about this just in terms of reflective stability, a reflectively stable agent has to never have a conflict between what its future self wants and what its past self wants. You can only achieve this by having a single agent at the center of control across all time. This center of control has to be simple enough that the agent can follow it in the early time steps, but the agent has to keep following it at the late time steps too. Thus, the agent in the late time steps has to be a powerful agent optimizing on behalf of this simple center of control.
# 10/28/17 Development Update: New editor framework (markdown support!), speed improvements, style improvements

Here is a quick summary of the changes I just pushed:

*   We now have a new editor framework (we moved from ory-editor to draft-js-plugins). The primary reason for this was performance, though we also had a few other problems with the editor. Please let me know in case anything breaks, but overall I expect things to be a lot more stable now.

*   With the move to the new framework comes markdown support. So feel free to use markdown in any of the text fields from now, it should be automatically translated into its rich-text equivalent (you can insert block-level images via the Toolbar and inline-images via markdown)

*   We significantly decreased our bundle size with this update, though I still have some more cleaning up to do. The editor was the biggest part of the rework that I was blocked on, so wrapping up the performance improvements should now be a lot faster
*   We also made some style improvements and navigation improvements to the sequences and collection navigation. Though we are still wrapping up some stuff in that domain, so expect more in the coming days.

**Edit:** I also made some changes to the font and typography, as you might have noticed. I actually like this font (Garamond Premiere Pro) less, but the other one had rendering issues on Windows machines, so here we are. I will adjust the page piece by piece to work better with the new font, and in general add a bunch more font and typography polish soon.

On that note, I also slightly restructured the frontpage to have a bit of a less chaotic structure and be easier to skim. This cost us some vertical space, but I think is worth the cost in structure and getting rid of that weird card-hover interaction.
# Competitive Truth-Seeking

  

In many domains, you can get better primarily by being correct more frequently. If you’re managing a team, or trying to improve your personal relationships, it’s very effective to improve your median decision. The more often you’re right, the better you’ll do, so people often implicitly optimize their success rate.

But what about competitive domains, like prediction markets, investing, or hiring?

One of the most common mistakes I see people make is trying to apply non-competitive intuitions to competitive dynamics. But the winning strategies are _very_ different! It is not enough to be right – others must be wrong. Rather than having a high success rate across all situations, you want to find an edge in some situations, and bet heavily when you find a mistake in the consensus.

For example, say you’re a startup trying to hire software engineers, and your process finds two excellent candidates – Alex and Bob. You make both of them offers at market rate for excellent software engineers. Alex knows how to interview well, so every company believes (correctly) that he’s excellent, while Bob is a bad interviewer, and most of your competitors think he’s merely good. As a result, there might be a 10% chance Alex accepts your offer in this situation, and a 50% chance Bob does.

Assuming a similar rate of Alexes and Bobs in the world, this means your process should be optimized almost entirely for finding Bobs – if you miss a few Alexes to find another Bob, that’s totally worth it! But from the outside, this will look crazy – you’re passing up obviously great candidates in order to specifically find people who aren’t obviously great.

Furthermore, this means that copying mainstream interviewing strategies is one of the worst things you can do – if you only get points for beating consensus predictions, then matching them will get you a 0.

The next time you find yourself thinking about one of these situations, try to figure out whether it’s competitive or non-competitive, and whether you should be optimizing for median predictions or edge.
# Announcing the AI Alignment Prize


Stronger than human artificial intelligence would be dangerous to humanity. It is vital any such intelligence’s goals are aligned with humanity's goals. Maximizing the chance that this happens is a difficult, important and under-studied problem.

To encourage more and better work on this important problem, we (Zvi Mowshowitz and Vladimir Slepnev) are announcing a $5000 prize for publicly posted work advancing understanding of AI alignment, funded by [Paul Christiano](https://paulfchristiano.com/).

This prize will be awarded based on entries gathered over the next two months. If the prize is successful, we will award further prizes in the future.

The prize is not backed by or affiliated with any organization.

Rules
-----

Your entry must be published online for the first time between November 3 and December 31, 2017, and contain novel ideas about AI alignment. Entries have no minimum or maximum size. Important ideas can be short!

Your entry must be written by you, and submitted before 9pm Pacific Time on December 31, 2017. Submit your entries either as links in the comments to this post, or by email to apply@ai-alignment.com. We may provide feedback on early entries to allow improvement.

We will award $5000 to between one and five winners. The first place winner will get at least $2500. The second place winner will get at least $1000. Other winners will get at least $500.

Entries will be judged subjectively. Final judgment will be by Paul Christiano. Prizes will be awarded on or before January 15, 2018.

What kind of work are we looking for?
-------------------------------------

AI Alignment focuses on ways to ensure that future smarter than human intelligence will have goals aligned with the goals of humanity. Many approaches to AI Alignment deserve attention. This includes technical and philosophical topics, as well as strategic research about related social, economic or political issues. A non-exhaustive list of technical and other topics can be found [here](https://docs.google.com/document/d/1bBspyL_S43gWjNtfcE_iQ5A_AT6snk8lmJD291_9_oo/edit).

We are **not** interested in research dealing with the dangers of existing machine learning systems commonly called AI that do not have smarter than human intelligence. These concerns are also understudied, but are not the subject of this prize except in the context of future smarter than human intelligence. We are also **not** interested in general AI research. We care about AI alignment, which may or may not also advance the cause of general AI research.

(Addendum: the results of the prize and the rules for the next round have now been [announced](https://www.lesserwrong.com/posts/4WbNGQMvuFtY3So7s/announcement-ai-alignment-prize-winners-and-next-round).)
# Modesty and diversity: a concrete suggestion

In online discussions, the number of upvotes or likes a contribution receives is often highly correlated with the social status of the author within that community. This makes the community less epistemically diverse, and can contribute to feelings of groupthink or [hero worship](http://effective-altruism.com/ea/1g3/why_how_to_make_progress_on_diversity_inclusion/).

Yet both the author of a contribution and its degree of support contain [bayesian evidence](http://thomas-sittler.github.io/authority/)about its value, arguably an amount that [should overwhelm](http://effective-altruism.com/ea/1g7/in_defence_of_epistemic_modesty/) your own inside view.

We want each individual to invest the socially optimal amount of resources into critically evaluating other people’s writing (which is higher than the amount that would be optimal for individual epistemic rationality). Yet we also all and each want to give sufficient weight to authority in forming our all-things-considered views.

As Greg Lewis writes:

> The distinction between ‘credence by my lights’ versus ‘credence all things considered’ allows the best of both worlds. One can say ‘by my lights, P’s credence is X’ yet at the same time ‘all things considered though, I take P’s credence to be Y’. One can form one’s own model of P, think the experts are wrong about P, and marshall evidence and arguments for why you are right and they are wrong; yet soberly realise that the chances are you are more likely mistaken; yet also think this effort is nonetheless valuable because even if one is most likely heading down a dead-end, the corporate efforts of people like you promises a good chance of someone finding a better path.

Full blinding to usernames and upvote counts is great for critical thinking. If all you see is the object level, you can’t be biased by anything else. The downside is you lose a lot of relevant information. A second downside is that anonymity reduces the selfish incentives to produce good content (we socially reward high-quality, civil discussion, and punish rudeness.)

I have a suggestion for capturing (some of) the best of both worlds:

*   first, do all your reading, thinking, upvoting and commenting with full blinding
*   once you have finished, un-blind yourself and use the new information to

*   form your all-things-considered view of the topic at hand
*   update your opinion of the people involved in the discussion (for example, if someone was a jerk, you lower your opinion of them).

To enable this, there are now two [user scripts](https://lifehacker.com/tag/user-scripts) which hide usernames and upvote counts on (1) [the EA forum](https://userstyles.org/styles/150270/effective-altruism-forum-anti-kibitzer) and (2) [LessWrong 2.0](https://userstyles.org/styles/150328/lesswrong-2-0-anti-biaser). You’ll need to install the Stylish browser extension to use them.

Cross-posted [here](https://thomas-sittler.github.io/blinding/) (clicking the link will unblind you!).
# Bet Payoff 1: OpenPhil/MIRI Grant Increase

_takes a deep breath_

It has come to one of those most special times. A time when a long-term bet is resolved.

Gregory Lewis (aka [Thrasymachus](http://lesserwrong.com/user/thrasymachus)) made a bet with me that the Open Philanthropy Project would not report an increased confidence in MIRI's impact, nor an increased grant size in the next 2 years.

And today, just 4 months later, the Open Philanthropy Project has announced (early) a [3.75 million dollar grant](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-general-support-2017) to be spread over 3 years (that's half of MIRI's budget each year; it may also be increased if MIRI's budget increases), and that this was in large part due to a significant update of their views on MIRI after a very positive review of MIRI's work on Logical Uncertainty by an external Machine Learning expert.

The initial bet and wording is at my (4 month old) facebook post [here.](https://www.facebook.com/ben.a.pace/posts/10209722845484959)

* * *

Crossposted from [Facebook](https://www.facebook.com/ben.a.pace/posts/10210558984827920?pnref=story)
# Reading Papers in Undergrad

I went to school for electrical engineering, and in my capstone electromagnetic theory course I encountered a problem that I needed to read papers to learn about. I asked my professor for his advice on _how_ to read them, owing to my lack of expertise, and he recommended the following:

  

**Don't** read the abstract. This is written by experts for other experts, so they can quickly see if it is relevant to their work. It will provide no useful information, and may intimidate or confuse you.

  

**Do** read the introduction. This will contain an overview of experiment, and most importantly describe some of the background and motivation for the experiment. As a non-expert, this context is at a premium. If the introduction does not suggest there will be useful information, you can stop here.

  

**Skip** the body of the paper. The methods and construction of the work are usually not the goal of students, and this part is the most dense and requires the greatest expertise.

  

**Do** read the conclusion. This is usually the meat of what a student is looking for - what do experts think, in their own words, about their work. With this information in hand, you can decide whether you need to go deeper into the paper or have what you need. If you decide to go deeper:

  

**Return** to the body of the paper. This is especially useful if you are greatly surprised by the conclusion, or if you need to be able to reference images or graphs in an informed way. Even as a non-expert you can profitably think about how the conclusion follows from the steps they have laid out. It is naturally required if you want to attempt to duplicate the experiment.

  

I have had pretty good success with this method, even in subjects for which I have less than undergrad expertise. I will add two additional steps I found necessary, both obvious:

  

**Reread** the paper. As you follow up on the problem with additional reading, you will gain important new perspectives and context that led to missing things the first pass.

  

**Follow-up** the references. The assumptions about how diligent readers are with following through vary; this can make something aggravatingly opaque suddenly clear because it was covered well in the reference, or reveal a gaping hole the authors failed to address.

  

Anecdotally, the problem I was thinking about was the electrical signature of cancer. I had found a few modern references to work done on the subject in animals, but couldn't lay hold of work on humans for the life of me. Finally on the third pass over a survey paper covering related work I saw an obscure reference, and following through with that led me to all of the work which had been done on humans previously.
# List of civilisational inadequacy

In the spirit of experimentation I want to have a thread where we list examples of gross civilizational inadequacy.

There's subcomment for meta comments on the thread, otherwise I encourage you to have one top-level comment per example of civilizational inadequacy.
# Examples of Mitigating Assumption Risk

One of the subskills mentioned in [Eliezer's Security Mindset post](https://www.lesserwrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia) is mitigating assumption risk–that is, the risk of losing utility because some of your assumptions are wrong. There are two main ways to do this:

1.  Gain more information about whether your assumptions hold
2.  Make the assumption irrelevant (such as the hashing passwords example)

Here are a bunch more examples:

*   Repeating back what someone said in your own words, to check understanding
*   Adding a margin of safety when estimating how much load a bridge can bear
*   Using Statistical models that make fewer assumptions, or have fatter tails
*   Exposing your work to attack in low-risk situations, such as comedians testing new material in small clubs, or Netflix's Chaos Monkey
*   Emphasizing fast adaption to unexpected circumstances over better forecasting
*   Putting spare capacity in steps in your process that aren't the bottleneck
*   Testing code frequently while refactoring to check functionality doesn't unintentionally change
*   Doing an analysis in different ways on different datasets, and only trusting them when the conclusions match
# DeepMind article: AI Safety Gridworlds

DeepMind authors present a set of toy environments that highlight various AI safety desiderata. Each is a 10x10 grid in which an agent completes a task by walking around obstacles, touching switches, etc. Some of the tests have a reward function and a hidden 'better-specified' reward function, which represents the true goals of the test. The agent is incentivized based on the reward function, but the goal is to find an agent that does not overfit it in a way that clashes with the hidden function.

I think this paper is relatively accessible to readers outside the machine learning community.
# Show LW: Diaspora Project Map and Preregistration Database

Hi everyone,

I and a handful of other folks have been compiling a [list of different LessWrong Diaspora projects](https://namespace.obormot.net/Main/DiasporaProjectMap). I started it after someone asked me how they could volunteer for LessWrong and I didn't have a good answer for them.

It's optimized for people who are researching what sort of things have already been done in the community, or are looking for a place where they can help. The basic criteria for eligibility are:

*   Being something I'm sure is a 'real' Diaspora project. i.e, started by a credible person who is a member of the Diaspora. (Membership is of course a little fuzzy, err on the side of inclusion.)
*   Either a web presence or contact information.

Suggestions for additions should go in the comment box on that page. (I'll also accept them in this thread, but understand I'll probably only check back on it for the first three days or so.)

We've also put together [a preregistration database.](https://namespace.obormot.net/Rdb/Project) A preregistration database solicits plans for projects and experiments before their results are known. It helps with the problem where no one wants to talk about something that didn't work. Ideally it becomes part of a framework for helping people put together solid projects.

Right now that framework includes a [Project Litmus Test](https://namespace.obormot.net/Main/ProjectLitmusTest) that's meant to help people who want to do something not fail from the outset through poor planning. The current question set is based on what we think are common failure modes for Diaspora projects, but over time I'd like to refactor it based on empirical interaction with peoples real ideas. In the best case scenario a workshop would be developed that helps people go from concept to execution.

My hope is that such a framework could put a real dent in [the chronic lack of coordination](http://www.greaterwrong.com/posts/XQWLq4DmrnKj9DPYC/straw-hufflepuffs-and-lone-heroes) that you've already been told about a thousand times.

[The Map](https://namespace.obormot.net/Main/DiasporaProjectMap)

[The Preregistation Database](https://namespace.obormot.net/Rdb/Project)
# 12/12/2017 Update: Creating Sequences

Over the past few weeks, we've been pushing updates that improve LW 2.0's Sequence functionality, as well as cleaning up some of the less well known sequences.

Those updates are now complete and ready for general use. Highlights include:

The Library
-----------

If you click on the "Recommended Reading" title on the front page, or on "The Library" in the main menu, you'll be taken to [our new Library page](https://www.lesserwrong.com/library). This includes sections for our Core Reading, Curated Sequences, and Community Sequences.

**Core Reading** is essential content that has stood the tests of time, that users are generally expected to have read.

**Curated Sequences** are additional high quality essays that the admins have decided to feature.

**Community Sequences** are where most new sequences will appear. (Currently sorted by "newest first", although we'll probably update the sorting mechanisms soon)

Creating New Sequences
----------------------

Under the "Community Sequences" header, you'll see a "Create new sequence" button. This is deliberately somewhat hard-to-find, since we don't want new users to immediately start creating sequences willy nilly.

To create a sequence, you'll start by giving it a name, description, and images for the Banner and Thumbnail. It's recommend that the Banner and Thumbnail be the same picture (although perhaps cropped differently so that they look good at large and small scales).

Banner images must be at least 1600 pixels wide. Eventually we'll add some tools to help users find appropriate images. Meanwhile, if you need help finding an appropriate banner, you can ping us and we'll help you out. (In general, a good tip is to go to Google Image search, click "tools" tab, and restrict results to images whose _size_ is at least 2 MP, and whose _usage rights_ are set to "noncommercial re-use)

After clicking the "create" button, you'll be able to add and change the ordering of posts.

User Profile Sequences
----------------------

Once you've created at least one sequence, your User Profile page will gain a Sequences section. This will list your sequences, and includes another "create sequence" button.

You can see an example of this at Luke Muehlhauser's [user page](https://www.lesserwrong.com/users/lukeprog).

Happy Sequencing!
-----------------

Creating better sequence tools is a core element of the LW 2.0 strategy. Some of our next goals are to improve your ability to keep track of which sequences you're currently reading (and for longer multi-part sequences, automatically suggesting the next sequence of a book when you finish)

Have fun creating new content!
# Oracle paper

Available on the arXiv, my paper on [two types of Oracles](https://arxiv.org/abs/1711.05541) (AIs constrained to answering questions only), and how to use them more safely.

> An Oracle is a design for potentially high power artificial intelligences (AIs), where the AI is made safe by restricting it to only answer questions. Unfortunately most designs cause the Oracle to be motivated to manipulate humans with the contents of their answers, and Oracles of potentially high intelligence might be very successful at this. Solving the problem, without compromising the accuracy of the answer, is tricky. This paper reduces the issue to a cryptographic-style problem of Alice ensuring that her Oracle answers her questions while not providing key information to an eavesdropping Eve. Two Oracle designs solve this problem, one counterfactual (the Oracle answers as if it expected its answer to never be read) and one on-policy (limited by the quantity of information it can transmit).
# Youtube channel devoted to the art of rationality

For quite some time I've considered creating a Youtube channel devoted to rationality.

I live in Italy, and the political situation here bears some similarities with the US': we've seen the rise of populist figures accompanied by the debate around fake news and post-truth, usually created with the purpose of bringing more votes to those politicians.

When a fake news is exposed, the standard answer is deploring the lack of critical thinking, and blaming school for failing to teach it. But, as far as I know, schools can only teach programs that are approved by the Department of Education, which 'institutionalize' living intellectual efforts like science and history. So far, no one has institutionalized critical thinking. In a sense, you could say that it cannot be completely institutionalized.

Sometimes, it's expected that philosophy classes teaches rationality, or at least thatteaches the pupils how to think. I know that LW users will appreciate the irony.

Anyway, philosophy is taught only in high school, and especially in poorer area not enough people reach that level of education, and is only taught within some 'orientation' (here in high school you cannot compose your courses as you wish, you have 'pre-packaged' sets of courses with a scientific or humanistic or artistic orientations).

In the end, the situation is that everybody is complaining about the lack of rationality teaching, but so far no-one has put himslef at creating such content.

Cue me.

I've been a member of LW since 2010, and I've always been interested in rationality, philosophy and AI. I'm not the smartest guy, so it's quite useless for me to be in academia or other research institution, but I've been told repeatedly that I'm good at explaining complicated things in lay terms. This seems a perfect combination of features for me to create content, aimed at the Italian situation and the Italian speaking audience, that tries to popularize rationality, Bayesianism and critical thinking.

With this post I would like to call for topics. What topic could I treat and how?

I already thought about something: locating rationality in the midst of and in contrast with other approaches; a non-technical survey of Bayesian probability; excercise in critical thinking (discussing a hot topic and trying to keep your identity small); atheism; the principle and crisis of science.

What do you think?
# I Am not a Tax Lawyer

  

I am really not a tax lawyer. This will become obvious soon enough, but it’s important for my readers to keep in mind from the very top. Still, [this spoke to my heart](https://www.bloomberg.com/view/articles/2017-12-08/bitcoin-swings-and-tax-glitches):

> I sometimes think I missed my calling as a tax lawyer. Tax lawyers, if they are any good, approach the world with a childlike sense of wonder. Someone (Congress) has given them a magical toy (the tax code) to play with, a toy whose possibilities are limited only by their imagination. And their job is to come up with new games to play with it, new combinations to try, new solutions to the riddles hidden in the code.  
> _– Matt Levine_

Well, Congress gave Americans a new toy, and so far all rationalists / effective altruists have done is [whine about it](http://slatestarcodex.com/2017/12/05/the-tax-bill-compared-to-other-very-expensive-things/). I actually think that Scott’s thinking about the tax plan is very confused. The tax code is not a _purchase_ of $100 billion, it’s better thought of as enacting a _transfer_ and establishing a set of _incentives_. The _transfer_ is mostly from people who will pay taxes in the future (young Americans) to people who owned American stocks in 2017 (older and richer Americans). This part isn’t great. The _incentive_ is for people and corporations to make money in the US. This part isn’t terrible.

We can compare this to Bernie Sanders’ free college plan, which Scott thinks is better. Subsidizing college would transfer money from taxpayers to colleges, which is pretty bad. It will also incentivize marginal students to waste more precious years in colleges while making the zero-sum game of credentialism and signaling ever more destructive, this part is really fucking bad.

The tax plan is poorly thought out, unfair to blue states, and inefficient. But it’s not a catastrophe – it’s not even as bad as [free college](http://slatestarcodex.com/2015/06/06/against-tulip-subsidies/). But it doesn’t matter how bad we think the tax plan is, the plan is here to stay (for at leat a few years). The important thing to focus on now is **how do we exploit it?**

If we’re playing a board game and you munchkin the rules, I can complain that you’re following the letter of the law but not the spirit. But the government can’t complain – it **writes **the rules, and it does so with the full expectation that every citizen will do their damnedest to exploit them as they are written. If we’re worried that greedy plutocrats get most of the gains from gaming the system, that just means that good folk like us need to step up and do our fair share of cheating optimizing.

[Continue reading the 5 schemes for tax-efficient donation on Putanumonit](https://putanumonit.com/2017/12/23/not-a-tax-lawyer/)
# The expected value of the long-term future

I wrote an article describing a simple model of the long-term future. Here it is:

*   in [PDF](https://thomas-sittler.github.io/ltf-paper/longtermfuture.pdf) format
*   in [tex](https://thomas-sittler.github.io/ltf-paper/longtermfuture.tex) format

Summary:

> A number of ambitious arguments have recently been proposed about the moral importance of the long-term future of humanity, on the scale of millions and billions of years. Several people have advanced arguments for a cluster of related views. Authors have variously claimed that shaping the trajectory along which our descendants develop over the very long run (Beckstead, 2013), or reducing extinction risk, or minimising existential risk (Bostrom, 2002), or reducing risks of severe suffering in the long-term future (Althaus and Gloor, 2016) are of huge or overwhelming importance. In this paper, I develop a simple model of the value of the long-term future, from a totalist, consequentialist, and welfarist (but not necessarily utilitarian) point of view. I show how the various claims can be expressed within the model, clarifying under which conditions the long-term becomes overwhelmingly important, and drawing tentative policy implications.
# Why did everything take so long?

One of the biggest intuitive mysteries to me is how humanity took so long to do anything.

Humans have been ‘behaviorally modern’ for about [50 thousand years](https://en.wikipedia.org/wiki/Timeline_of_human_evolution#Homo). And apparently didn’t invent, for instance:

*   rope [until 28 thousand years ago](https://en.wikipedia.org/wiki/Timeline_of_historic_inventions#Paleolithic).
*   the wheel [until at least 4000BC](https://en.wikipedia.org/wiki/Timeline_of_historic_inventions#Neolithic)
*   writing [until 3000BC](https://en.wikipedia.org/wiki/Timeline_of_historic_inventions#Neolithic)
*   woodblock printing [until 200AD](https://en.wikipedia.org/wiki/Timeline_of_historic_inventions#3rd_century)

This kind of thing seems really weird introspectively, because it is hard to imagine going a whole lifetime in the wilderness without wanting something like rope, or going a whole day wanting something like rope without figuring out how to make something like rope. Yet apparently people went for about a thousand lifetimes without that happening.

Some possible explanations:

1.  Inventions are usually more ingenious than they seem. [LiveScience](https://www.livescience.com/18808-invention-wheel.html) argues that it took so long to invent the wheel because “The tricky thing about the wheel is not conceiving of a cylinder rolling on its edge. It’s figuring out how to connect a stable, stationary platform to that cylinder.” I feel like that would explain why it took a month rather than a day. But a couple of thousand lifetimes?
2.  Knowing what you are looking for is everything. If you sat a person down and said, “look, how do you attach a stationary platform to a rolling thing?” they could figure it out within a few hours, but if you just give them the world, they don’t think about whether a stationary platform attached to a rolling thing would be useful, so “how do you attach a stationary platform to a rolling thing” doesn’t come up as a salient question for a couple of thousand lifetimes.
3.  Having concepts in general is a big deal, and being an early human who had never heard of any invention was a bit like being me when I’m half asleep.
4.  Everything is always mysteriously a thousand times harder than you might think. Consider writing a blog post. Why haven’t I written a blog post in a month?
5.  Others?
# Have you felt exiert yet?

Pre-adolescent children haven't felt strong lust yet. Those of us who've avoided strong pain are also missing an experience of that affect. Nostalgia can come up very early, but does require a bit of living first. Depression can strike at any age.

So, in general, there are emotions and feelings that people are capable of feeling, in the right circumstances, but that some people have not yet felt.

As a thought experiment, I'd thus like to introduce the emotion of exiert, which no human being has yet felt, because it's only triggered by (very) unusual experiences. Maybe it's something that can only be felt after a decade of weightlessness, or after the human brain has adapted to the living for a long time within the atmosphere of a gas giant.

Let's assume that exiert has some survival uses - it's helpful in navigating the gas giant's ever-changing geography, say - and it can load onto both positive and negative affect (just as confusion and surprise can). Assume also that experiencing exiert can have impact on people's aethetic preferences - it can cause them to like certain colour schemes, or open them to different types of pacing and tone in films or theatre productions.

In the spirit of trying to deduce [what human values really are](https://www.lesserwrong.com/posts/KCg7NeKQ7MycXWpYd/our-values-are-underdefined-changeable-and-manipulable), the question is then: is the AI overriding human preferences in any of the following situations:

1.  The AI premeptively precludes the experience of exiert, so that we won't experience this emotion even if we are put in the unusual circumstances.
2.  Suppose that we wouldn't "naturally" experience exiert, but the AI acts to ensure that we would (in those unusual circumstances).
3.  The AI acts to ensure that some human that had not yet experienced strong lust, nostalgia, or extreme pain, could never experience that emotion.
4.  Suppose some human would not "naturally" experience strong lust, nostalgia, or extreme pain, but the AI acts to ensure that they would.
5.  The AI acts to ensure that some human experiences exiert, but that this is triggered by normal circumstances, rather than unusual ones.
# Rationality: Abridged

This was originally planned for release around Christmas, but our old friend Mr. Planning Fallacy said no. The best time to plant an oak tree is twenty years ago; the second-best time is today.

I present to you: _Rationality Abridged --_ a 120-page nearly 50,000-word summary of "Rationality: From AI to Zombies". Yes, it's almost a short book. But it is also true that it's less than 1/10th the length of the original. That should give you some perspective on how massively long R:AZ actually is.

As I note in the Preface, part of what motivated me to write this was the fact that the existing summaries out there (like the ones on the LW Wiki, or the _Whirlwind Tour_) are too short, are incomplete (e.g. not summarizing the "interludes"), and lack illustrations or a glossary. As such, they are mainly useful for those who have already read the articles to quickly glance at what it was about so they could refresh their memory. My aim was to serve that same purpose, while **also** being somewhat more detailed/extensive and including more examples from the articles in the summaries, so that they could also be used by newcomers to the rationality community to understand the key points. Thus, it is essentially a heavily abridged version of R:AZ.

[Here is the link to the document.](https://drive.google.com/file/d/1xwvQ_lxhxCoN6T92VX6sKeDWsMk-fxFb/view?usp=sharing) It is a PDF file (size 2.80MB), although if someone wants to convert it to _.epub_ or _.mobi_ format and share it here, you're welcome to.

There is also a text copy at my brand new blog: [perpetualcanon.blogspot.com/p/rationality.html](http://perpetualcanon.blogspot.com/p/rationality.html)

I hope you enjoy it.

(By the way, this is my first post. I've been lurking around for a while.)
# LW Update 1/5/2018 – Comment Styling

We've just pushed an update with a few stylistic tweaks, a few small bug fixes, and most notably, an overhaul on the comment styling.

The intent here was to make it easier to skim comments and keep track of branching threads. This is a bit of an experiment, and the current approach is that we're treating the comment section as conceptually different from the main post section.

This [post has moderately deep comment trees](https://www.lesserwrong.com/posts/cw5voHpqoZMyz4DWg/cash-transfers-are-not-necessarily-wealth-transfers) that can showcase it in action.

Bugs that we could easily remember fixing:

*   KarmaThreshold for RSS feed should work now
*   Auto-styled links no longer lie to you
*   Meta posts show up in the Daily feed
*   Magical Sorting on the front page now favors Front Page posts and Curated posts slightly (with +10 and +20 effective karma,