Harley-ml/lesswrong huggingface dataset:
# Shared Cache is Going Away

Browsers historically have had a single HTTP Cache. This meant that if www.a.example and www.b.example both used cdn.example/jquery-1.2.1.js then JQuery would only be downloaded once. Since it's the same resource regardless of which site initiates the download, a single shared cache is more efficient. \[1\]

Unfortunately, a shared cache enables a privacy leak. Summary of the simplest version:

*   I want to know if you're a moderator on www.forum.example.
*   I know that only pages under www.forum.example/moderators/private/ load www.forum.example/moderators/header.css.
*   When you visit my page I load www.forum.example/moderators/header.css and see if it came from cache.

Versions of this have been around for a while, but in March 2019 Eduardo Vela disclosed a [way to make it much more powerful and reliable](https://sirdarckcat.blogspot.com/2019/03/http-cache-cross-site-leaks.html). Browsers are responding by partitioning the cache ( [Chrome](https://www.chromestatus.com/feature/5730772021411840), [Firefox](https://bugzilla.mozilla.org/show_bug.cgi?id=1536058); Safari [already had](https://bugs.webkit.org/show_bug.cgi?id=110269)). \[2\] It's not clear from me reading the bugs when it will launch, but it does sound soon. \[3\]

What does this mean for developers? The main thing is that there's no longer any advantage to trying to use the same URLs as other sites. You won't get performance benefits from using a canonical URL over hosting on your own site (unless they're on a CDN and you're not) and you have no reason to use the same version as everyone else (but staying current is still a good idea).

I'm sad about this change from a general web performance perspective and from the perspective of someone who really likes small independent sites, but I don't see a way to get the performance benefits without the leaks.

  
\[1\] When I worked on [mod_pagespeed](https://www.modpagespeed.com/), rewriting web pages so they would load faster, we had an opt-in feature to [Canonicalize JavaScript Libraries](https://www.modpagespeed.com/doc/filter-canonicalize-js).

\[2\] I was curious if this had launched yet so I made a pair of test pages and tried it out in WebPageTest for [Chrome Canary](https://webpagetest.org/result/191101_KK_ca05481e79950902c9491ac7e7ff9a6c/) and [Firefox Nightly](https://webpagetest.org/result/191101_VD_b829e9699c17349bce52b663474bfce1/) but it's not out yet. I used a WPT script consisting of:

navigate https://www.trycontra.com/test/cache-partition
navigate https://www.bidadance.org/test/cache-partition

\[3\] Firefox's [bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1536058) is marked "fixed" and "Milestone: mozilla70", but I have Firefox 70.0.1 and it doesn't seem to be enabled:

[![](https://www.jefftk.com/firefox-not-cache-partitioning.png)](https://www.jefftk.com/firefox-not-cache-partitioning-big.png)

Perhaps this is just the code change and they still need a flag flip? I don't know how Firefox does this.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100119293431842), [lesswrong](https://lesswrong.com/posts/La4v9GBcqx6tij2FX)*
# Speaking up publicly is heroic

*Content warning: discussion of abuse.*

I was recently in a discussion of safety, "me too", and responding to harmful behavior where someone wrote, paraphrased:

> Discussing things like this, publicly and in writing, is dangerous. Posting negative things about people could get you sued for libel, or even physically assaulted if the person takes it poorly enough. Why would you put yourself at risk, just to get something off your chest?

While they're right that speaking up is risky, the idea that people who make public posts are doing it lightly doesn't match what I've seen. Most people who post know the risk, but decide to go ahead for important reasons including "no one else should have to go through this" and "if people know bad behavior will come back to bite them they'll behave better."

In November 2017 someone in one of my communities posted publicly about someone who had abused them, and then in September 2018 two people in another of my communities posted about a different person. \[1\] While these two abusers were different in a series of ways, they had a lot in common: they were relatively popular, they used age, inexperience, and intoxicants to take advantage of people, they had abused a series of people, and there were [whispered warnings](https://en.wikipedia.org/wiki/Missing_stair) but not everyone heard or believed them.

In both cases, the public posts completely changed the situation by pulling all of this into common knowledge. The abusers were rightly ejected and communities started working on figuring out how they had let things get to this point.

We need to build a world where this is not needed, and where we have good social systems for handling harmful behavior. I don't know what this should look like yet, however, and all the proposals I've seen have good and bad parts. In my corner of the world I've been working on this with [BIDA's safety committee](https://www.bidadance.org/safety/), and I'm glad that communities are generally taking this more seriously. We also need to handle this on a personal level, supporting friends who've been hurt and holding others accountable when they hurt people. But while stopping abusers should not require this level of personal risk, as long as it does I have enormous respect for people who decide to take that risk and speak up.

  
\[1\] I'm not linking the two here, even though both are world-readable, because they've served their purposes and I'm not trying to put more attention on those two cases in particular.
# Open & Welcome Thread - November 2019

*   If it’s worth saying, but not worth its own post, here's a place to put it.

*   You can also make a [shortform post](http://www.lesslong.com/).

*   And, if you are new to LessWrong, here's the place to introduce yourself.

*   Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Ways that China is surpassing the US

[Edit: Changed the post title from the original article title to something more meaningful.]

I came across this article today, and I have to agree with it strongly based on my own recent trip to China. The update it triggered for me is the realization that China is genuinely doing better than the US on many fronts, most importantly on *governance*. How/why did that happen? Did anyone or any political theory predict this ahead of time? (In case it's not clear, this is not meant to be a rhetorical question. I'm surprised and confused and am wondering if someone or some theory can offer an explanation.) What does it imply for things like AI governance and global coordination on x-risks?

Below I'll quote two sections that summarize most of the article. Click the link above for the full text.

---

China is changing in a deep and visceral way, and it is changing fast , in a way that is almost incomprehensible without seeing it in person. In contrast to America’s stagnation, China’s culture, self-concept, and morale are being transformed at a rapid pace—mostly for the better.

For Americans, this transformation towards competence and prosperity should be the more worrying thing about China. We aren’t well-coordinated with China, and it is starting to surpass the U.S. in important ways. If we ignore this point by focusing only on the moral negatives, of which there are many, we risk underrating the competition until it’s too late, or failing to reflect on the importance of solving our own mounting problems of governance.

More optimistically, we should study Chinese development as a positive example of how to do better on the things that America has been fumbling recently: infrastructure, growth, industrial policy, and positive transformation of everyday life. America has been a shining beacon of this kind of development in the past, and can be again, if we’re willing to look closer at what China does right.

[...]

If I’m being honest, China’s success scares me. There is something deeply disconcerting about watching China surpass America in the ways it is. China is transforming fishing villages into major industrial cities, while we fail to build high-speed rail or new housing. How are we going to catch up?

Is it really as bad as it seems? I understand the instinct to avoid the topic, disbelieve it, and play it down. I’ve even had the instinct to stay quiet about China’s progress myself: I worry that no one will appreciate the reminder. Perhaps this is why we’ve heard so little about this aspect of things.

But if we’re going to build a good society in America, we have to face these things head-on.

In the U.S., we face an ongoing crisis of governance. We need to understand our own failures, and we need to grapple with unexpected demonstrations of success—even if they come from non-liberal societies.

China’s success challenges our implicit ideology and deep-seated assumptions about governance. It needs to be studied—not just to bring about better coordination, but because in its accomplishments, we may find important truths needed to bring about American revitalization.
# What AI safety problems need solving for safe AI research assistants?

In his [AI Safety “Success Stories”](https://www.alignmentforum.org/posts/bnY3L48TtDrKTzGRb/ai-safety-success-stories) post, Wei Dai writes:

>[This] comparison table makes Research Assistant seem a particularly attractive scenario to aim for, as a stepping stone to a more definitive success story. Is this conclusion actually justified?

I share Wei Dai's intuition that the Research Assistant path is neglected, and I want to better understand the safety problems involved in this path.

Specifically, I'm envisioning AI research assistants, built without any kind of reinforcement learning, that help AI alignment researchers identify, understand, and solve AI alignment problems.  Some concrete examples:

**Possible with yesterday's technology**: Document clustering that automatically organizes every blog post about AI alignment.  Recommendation systems that find AI alignment posts similar to the one you're reading & identify connections between the thinking of various authors.

**May be possible with current or near future technology**: An AI chatbot, trained on every blog post about AI alignment, which makes the case for AI alignment to skeptics or attempts to shoot down FAI proposals.  Text summarization software that compresses a long discussion between two forum users in a way that both feel is accurate and fair.  A NLP system that automatically organizes AI safety writings into a problem/solution table as I described in [this post](https://www.lesswrong.com/posts/znt3p9AGQDbYGf9Sy/the-problem-solution-matrix-calculating-the-probability-of).

**May be possible with future breakthroughs in unsupervised learning, generative modeling, natural language understanding, etc.**: An AI system that generates novel FAI proposals, or writes code for an FAI directly, and tries to break its own designs.  An AI system that augments the problem/solution table from [this post](https://www.lesswrong.com/posts/znt3p9AGQDbYGf9Sy/the-problem-solution-matrix-calculating-the-probability-of) with new rows and columns generated based on original reasoning.

What safety problems are involved in creating research assistants of this sort?  I'm especially interested in safety problems which haven't yet received much attention, and safety problems with advanced assistants based on future breakthroughs.
# Lite Blocking

There are people I would like to interact less with online: perhaps they post inane things, perhaps they pull comment threads off in bad directions, perhaps they make terrible arguments for things I agree with. The standard tools social networks offer for this sort of situation are:

*   Blocking: you can't see their stuff, they can't see your stuff, you can't interact.
    
*   Hiding: you don't see their stuff, or you see less of it. You can still interact.
    

I'd like another tool:

*   Lite blocking: as if they'd hidden you.
    

Social networks generally have far more things they could show you than you'll be able to look at. To prioritize they use inscrutable algorithms that boil down to "we show you the things we predict you're going to like". You can think of hiding as "dramatically lower the prediction that I would like seeing their stuff", and lite blocking as "dramatically lower the prediction that they would like seeing my stuff".

Lite blocking could be symmetrical or not, but the important thing to me is that the network would stop encouraging people to interact with me if I don't want those interactions.

Perhaps I should just block people? I'm glad blocking exists, and there are times when it's the right tool. But other times it's much too powerful:

*   Blocking is obvious, but with the level of "who knows why the things decided to show me what it did" lite blocking maintains plausible deniability. This is important for people you want to avoid interacting with but need to stay on good terms with for other reasons.
    
*   Blocking is too thorough. Maybe I don't like the way you tend to come into threads I host and derail them, but I'd still like you to be able to find our past discussions and reference them if intentionally seek them out.
    
*   Groups often have high thresholds for kicking people out, but an in-between level where someone would just see fewer group posts in their feed would be helpful in cases where the moderators think someone's posts are generally making the group worse.
    

Since lite blocking explicitly overrides the network's prediction about how much someone will like things, you could imagine a change to add it being difficult to get past launch review. It would probably look bad on the core metrics, with a decrease in estimated user satisfaction and engagement. But the metrics don't capture the ways "so and so isn't showing up in my discussions any more" would make others happier and improve their experience using the network.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100119961967092)*
# Normative reductionism

Here’s a concept that seems useful, but that I don’t remember ever hearing explicitly referred to (with my own tentative name for it—if it turns out to not already have one in some extensive philosophical literature, I might think more about whether it is a good name):

> *Normative reductionism*: The value of a world history is equal to the value of its parts (for some definition of relevant parts).

For instance, if two world histories only differ between time t and time t’, according to NR you do not need to know what happened at other times to evaluate them in full. Similarly, the value of Alice’s life, or the value of Alice enjoying a nap, depend on the nature of her life or the nap, and not for instance on other people’s lives or events that took place before she was born with no effect on her (unless perhaps she has preferences about those events or they involve people having preferences about her, but still the total value can be decomposed into the value of different preferences being fulfilled or not). Straightforward hedonistic utilitarianism probably implies normative reductionism.

My impression is that people have different intuitions about this and vary in how much they assume it, and that it mostly isn’t entirely aligned with other axes of ethical view, either logically or sociologically, though is related to them. So it seems maybe worth noting explicitly.
# Personal quality experimentation

Different people seem to have different strategies, which they use systematically across different parts of their lives, and that we recognize and talk about. For instance people vary on:

*   Spontaneity
*   Inclination toward explicit calculations
*   Tendency to go meta
*   Skepticism
*   Optimism
*   Tendency to look at the big picture vs. the details
*   Expressed confidence
*   Enacted patience

I don’t know of almost anyone experimenting with varying these axes, to see which setting is best for them, or even what different settings are like. Which seems like a natural thing to do in some sense, given the variation in starting positions and lack of consensus on which positions are best.

Possibly it is just very hard to change them, but my impression is that for at least some of them it is not hard to try, or to change them a bit for a short period, with some effort. (I have briefly tried making decisions faster and expressing more confidence.) And my guess is that that is enough to often be interesting. Also that if you effortfully force yourself to be more skeptical and it seems to go really well, you will find that it becomes appealing and thus easier to keep up and then get used to. 

I also haven’t done this much, and it isn’t very clear to me why. Maybe it just doesn’t occur to people that much for some reason. (It also doesn’t occur to people to choose their value of time via experimentation I think, a related suggestion I like, I think from Tyler Cowen a long time ago.) So here, I suggest it. Fun date activity, maybe: randomly reselect one personality trait each, and both try to guess which one the other person is putting on.
# Literature on memetics?

In certain circles, it's common to talk about memes as entities, in a metaphorical and handwaving way. But it seems to me that thinking rigorously and precisely about memes in the framework of "memes::genes, memeplexes::infectious agents, minds::hosts" actually has a lot of explanatory and predictive power. This kind of thing ought to be a tool in the rationalist toolkit. In the space of politics, for example, it can be very illuminating to strip away the content of an idea and analyze it purely as a replicator, or as a component of a larger replicator conferring a specific fitness advantage.

A cursory online search for existing literature on memetics reveals that the field is about as confused as you would expect. But perhaps I am going about my search in the wrong way. Is there another field that encompasses the kind of thing I am gesturing at here?
# New evidence on popular perception of "AI" risk

The Mozilla Foundation recently conducted a survey about AI perception. I participated after receiving an email invitation; I don't know if they recruited elsewhere. They just released a writeup of the results today, which can be found [here](https://foundation.mozilla.org/en/blog/we-asked-people-around-the-world-how-they-feel-about-artificial-intelligence-heres-what-we-learned/?utm_source=email&utm_medium=email&utm_campaign=2019engagement-en&utm_content=aisurveyreport&utm_term=5174656).

Mozilla seems to take a broad view of "AI", with a lot of weight given to current and near-term issues and less to potential fate-of-humanity issues. This is, of course, a different perspective from many in the LW crowd.

The popular perception is nonetheless potentially relevant to AI safety because it may represent the perception of some AI capabilities researchers. It may also be relevant to predicting political intervention in AI research.

Highlights:

*   51k sample size, 67k including partial submissions (see pdf linked from writeup)
*   overall, respondents were optimistic (i.e. not especially worried about risks)
*   most optimistic demographics were 19-24 years old, male, South Americans
*   only 10% self-reported being "well-educated" about AI, only 4% unfamiliar with term "AI"
*   respondents generally interested in learning more about AI

From link:

"24% of respondents said AI will make our lives better. 41% of respondents think AI will make our lives both better and worse. Only 10% of respondents think AI will only make our lives worse"

" Men (27%) are almost twice as optimistic as women (14%) that AI will make our lives better. Nearly half of South Americans (46%) are optimistic that AI is going to make their world better, making them the most positive region in the world. And young people 19 - 24 years old (35%) were the most likely age group to say AI will make the world better. Only 5% of this age group said they thought AI would make the world worse. "

*   "Respondents aged 19 - 44 were most likely to be “very interested” in learning more about AI (52%).
*   Respondents 65 and over were most likely to say they are somewhat interested in learning more about AI (64%)
*   South Americans (61%) and Africans (63%) said they were most interested in learning more about AI."
# Catalyst: a collaborative biosecurity summit

Applications for [Catalyst](https://catalystbiosummit.com/), a collaborative biosecurity summit, are [now open](https://docs.google.com/forms/d/e/1FAIpQLSfxMZ5qy0QcFYsV2ZZmglngk9znXhEEgq2Xt4-TmboQBzEm2A/viewform).

On February 22, 2020, at [The Laundry](http://thelaundrysf.com/) in San Francisco, a diverse group of synthetic biologists, policymakers, academics, and biohackers will come together to discuss one of the biggest emerging questions of the 21st century—how do we engineer a future enhanced by biotechnology and not endangered by it?

If you’d like to meet other individuals invested in the future of biotechnology, participate in forward-thinking conversations, or brainstorm solutions to open problems in biosecurity, [apply](https://docs.google.com/forms/d/e/1FAIpQLSfxMZ5qy0QcFYsV2ZZmglngk9znXhEEgq2Xt4-TmboQBzEm2A/viewform) by December 7th to secure a spot.
# For the metaphors

I make use of a lot of analogies, for instance ‘like dancing’ and ‘the ice skating thing’ are particular phenomena I often think about, and I get value from thinking about meta-ethics as if it were romance, or saving the world as if it were a party. I wonder if providing a variety of concrete experiences that other things might be analogized to is a big source of value from doing new things.

For instance, recently I took up knitting and I think there are things about it that my other experiences don’t have. For instance, I got some knitting patterns, and they have this very brief and utilitarian jargon, and a bunch of concepts, and I got a sense of this rich world of actionable and actioned knowledge about how to do a concrete thing, with much doing of it, which is pretty unlike other things I engage in, I am sorry to say. 

I was also struck by the experience of being able to take a relatively simple substance (wool) and turn it into a useful object of the kind one buys in a store (a hat, or it seems like it will be a hat). 

These things are of course what I expect in the abstract, but it is something else to experience things.

I’m not sure how these new experiences compare to the value I have had so far from the activity of knitting, but it seems like much more than the value of a generic hat, and I only have maybe a quarter of one of those.

My current guess is that filling out my repertoire of concrete intuitions about specific kinds of occurrences or relationships between things is pretty helpful.
# Pricing externalities is not necessarily economically efficient

> \[A\]s long as externalities exist and are not internalized via Pigouvian taxes, the result is inefficient. The inefficiency is eliminated by charging the polluter an emission fee equal to the damage done by his pollution. In some real world cases it may be difficult to measure the amount of the damage, but, provided that that problem can be solved, using Pigouvian taxes to internalize externalities produces the efficient outcome.  
>   
> That analysis was accepted by virtually the entire economics profession prior to Coase's work in the field. It is wrong—not in one way but in three. The existence of externalities does not necessarily lead to an inefficient result. Pigouvian taxes, even if they can be correctly calculated, do not in general lead to the efficient result. Third, and most important, the problem is not really externalities at all—it is transaction costs.
# Ethical experimentation

I [suggested](https://meteuphoric.com/2019/11/06/personal-quality-experimentation/) experimenting with different settings on personal characteristics that aren’t obviously good or bad. For instance, trying out being more or less perfectionistic for a day.

A particular variety of this that interests me is experimentation with different ethical principles, where opinion differs on which is correct. Both at levels of action (being a vegan for a week) and of abstract belief (being a virtue ethicist for a week).

I think this is a particularly non-obvious thing to do, because:

1.  You already have views on what is correct, and trading your good ethical principles for bad ethical principles seems unethical.
2.  Ethics seems like an area where experimentation is particularly unhelpful, being mostly about things outside of you that you don’t have direct access to, and also arguably inhabiting a separate realm that doesn’t interact with empirical facts. 

I think it is a good idea anyway. On 1, this is basically the same as the case for placebo controlled medical trials, assuming that the thing can actually help you be more ethical in the long run.

On 2, the main thing you have to go by on ethics is intuitions and arguments that are salient and moving to you. But people are notoriously bad at coming up with an unbiased selection of considerations to make salient on topics where they feel something, and it is easy to hear an argument and not really feel it. Actually trying to inhabit the different positions seems helpful for these.

I haven’t done this, but I have become a vegetarian for no great reason and in spite of my [argument](https://meteuphoric.com/2014/11/21/when-should-an-effective-altruist-be-vegetarian/) that it is not an effective use of effort, and then gone back to eating fish, and I found both things had pretty interesting effects on my intuitions about things and the arguments I thought about (possibly changing ethical positions for no good reason is especially good, because then your brain tries to make up its best reasons).

I was thinking of trying nihilism week soon, but then I got busy and maybe became a nihilist anyway, so we’ll see.
#  Indescribable

Some things can be described only via experience.
- Direct sensory experience (such as the color red)
- Foreign untranslatable words and phrases
- [Rasas](https://en.wikipedia.org/wiki/Rasa_(aesthetics))
- Certain meditative states (such as *kenshō* and *satori*)

Other things cannot be precisely described at all.
- Any particular noncomputable number

Indescribable things cannot be described in a finite number of words. That's because each one contains an infinite quantity of information. I don't mean they convey this information all at once (except for noncomputable numbers). Rather, they open up a new channel of information.

Opening up a new channel of information is mathematically equivalent to adding an input node to a neural network. This is a totally different process from training a machine learning system. When you train a neural network you adjust the weights of the connections between neurons to solve a problem. Adding hidden nodes amounts to basically the same thing. Adding an input node to a neural network unfolds a new dimension of the problem space thereby adding information for the network to work with. In other words, adding a new input node doesn't improve your solution to a problem at all. It makes the problem easier instead.

For example, the traditional Chinese method of teaching strategy involves memorizing ancient Taoist texts. This pedagogical technique is off the radar of modern MBA programs for reasons independent of its effectiveness. The "memorized passages of concise time-tested wisdom" input node is just missing.

You can't tell someone an indescribable thing but you can tell her where to look. You can tell someone to identify the color of blood, listen to a piece of music or stare silently at a blank wall until you see the essence of reality.

> The arts of war…cannot be ignored.
>
> ―first line of Sunzi's *The Art of War*

Where else should I look?
# Operationalizing Newcomb's Problem

The standard formulation of Newcomb's problem has always bothered me, because it seemed like a weird hypothetical designed to make people give the wrong answer. When I first saw it, my immediate response was that I would two-box, because really, I just don't believe in this "perfect predictor" Omega. And while it may be true that [Newcomblike problems are the norm](http://mindingourway.com/newcomblike-problems-are-the-norm/), most real situations are not so clear cut. It can be quite hard to demonstrate why causal decision theory is inadequate, let alone build up an intuition about it. In fact, the closest I've seen to a real-world example that made intuitive sense is [Narrative Breadcrumbs vs Grizzly Bear](https://sinceriously.fyi/narrative-breadcrumbs-vs-grizzly-bear/), which still requires a fair amount of suspension of disbelief.

So, here I'd like to propose a thought experiment that would (more or less*) also work as an actual experiment.

_A psychologist contacts you and asks you to sign up for an experiment in exchange for a payment. You agree to participate and sign all the forms. The psychologist tells you: "I am going to administer a polygraph (lie detector) test in which I ask whether you are going to sit in our waiting room for ten minutes after we finish the experiment. I won't tell you whether you passed, but I will give you some money in a sealed envelope, which you may open once you leave the building. If you say yes, and you pass the test, it will be $200. If you say no, or you fail the test, it will be $10. Then we are done, and you may either sit in the waiting room or leave. Please feel no obligation to stay, as the results are equally useful to us either way. The polygraph test is not perfect, but has so far been 90% accurate in predicting whether people stay or leave; 90% of the people who stay for ten minutes get $200, and 90% of those who leave immediately get $10."_

You say you'll stay. You get your envelope. Do you leave the building right away, or sit in the waiting room first?

Does the answer change if you are allowed to open the envelope before deciding?

*I don't know if polygraphs are accurate enough to make this test work in the real world or not.
# Can indifference methods redeem person-affecting views?

My uninformed paraphrase/summary of "the person-affecting view" is: "classical utilitarianism + indifference to creating/destroying people".

These views seem problematic (e.g. see [Hillary Greeves interview on 80k)](https://80000hours.org/podcast/episodes/hilary-greaves-global-priorities-institute/), and difficult to support.

Indifference methods (e.g. see [Stuart Armstrong's paper](https://arxiv.org/abs/1712.06365)) seem like they might be a way to formalize the person-affecting view in a rigorous way.

If we have a policy, we can always reverse engineer a corresponding reward function (see [our Reward Modelling agenda](https://arxiv.org/abs/1811.07871), bottom page 6).

So while there might still be highly counter-intuitive bullets that need to be bitten, this might provide a way of cashing out person-affecting views in a way that is mathematically coherent/consistent.

What do you think? Does it work?

And is that even an open problem, or an interesting result to people in ethics?
# An optimal stopping paradox

Consider an optimal stopping problem: a company at each time step grows by some constant, and has a certain probability of shutting down. You decide when to sell the company.

Since the math is cleaner in continuous time, we consider the continuous time. Then the company has a linearly increasing value βt, and an exponentially decaying survival curve e^(-αt).

Another framing of the paradox: Schrodinger wants to make a new record for the longest surviving cat, so he put a cat in the box with an atom that might decay and kill the cat, and waits. When should he open the box? 

Since at each moment in time, you face the exact same problem (linearly increasing reward, α-exponentially decaying survival rate), if you decide to wait at t=0, you would decide to wait forever, and thus receive no reward.

There are several possible replies to this paradox, none of which is satisfactory to me:

1. "This looks like St. Petersburg Paradox.". No, because at time t=0, the expectation is β/α^2. In fact, the payoff can grow faster than βt, such as like t^3, and it would still have finite expectation.
2. Claim that expectation maximization decision theory is flawed. This doesn't stop the procrastination. As long as your decision is purely based on the future, and your rational decision process is constant in time, you either immediately sell the company or never sell the company.
3. Try some kind of discounting, like exponential discounting. This doesn't stop the procrastination., since at any time, selling the company gives you 0 extra expected reward, and waiting gives you *some* positive extra expected reward, no matter how much you discount the future.
4. Claim that there should be a finite lifetime. You can't wait forever. If there is a finite lifetime, then the same decision analysis would tell you to procrastinate until the very end. This effectively is procrastinating forever. It does not converge to a reasonable finite waiting time as your lifetime goes to infinity.
5. Claim that one should stick to past decisions even when they don't make sense from a purely future-looking decision theory. Such decision theory seems to be
 just sweeping time-inconsistency under the rug, and I'm sure would suffer from serious paradoxes of their own.
6. Claim that there is no paradox, and procrastination is really the rational action. I'd not claim a strategy that guarantees 0 reward to be rational.

Option 5 seems at least to have some meaning to it. Sticking to it would mean that, for example, one would at t=0 decide to choose T to maximize βT e^(-αT), then at t=T really sell the company, even though it's irrational, conditional on the company still alive at t=T.
# What I’ll be doing at MIRI

_Note: This is a personal post describing my own plans, not a post with actual research content._

Having finished my internship working with Paul Christiano and others at OpenAI, I’ll be moving to doing research at MIRI. I’ve decided to do research at MIRI because I believe MIRI will be the easiest, most convenient place for me to continue doing research in the near future. That being said, there are a couple of particular aspects of what I’ll be doing at MIRI that I think are worth being explicit about.

First, and most importantly, this decision does not represent any substantive change in my beliefs regarding AI safety. In particular, my research continues to be focused around solving [inner alignment](https://arxiv.org/abs/1906.01820) for [amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd). My post on [relaxed adversarial training](https://www.alignmentforum.org/posts/9Dy5YRaoCxH9zuJqa/relaxed-adversarial-training-for-inner-alignment) continues to represent a fairly up-to-date form of what I think needs to be done along these lines.

Second, my research will remain public by default. I have discussed with MIRI their [decision to make their research non-disclosed-by-default](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/) and we agreed that my research agenda is a reasonable exception. I strongly believe in the importance of collaborating with both the AI safety and machine learning communities and thus believe in the need for sharing research. Of course, I also fully believe in the importance of carefully reviewing possible harmful effects from publishing before disclosing results—and will continue to do so with all of my research—though I will attempt to publish anything I don’t believe to pose a meaningful risk.

Third—and this should go without saying—I fully anticipate continuing to collaborate with other researchers at other institutions such as OpenAI, Ought, CHAI, DeepMind, FHI, etc. The task of making AGI safe is a huge endeavor that I fully believe will require the joint work of an entire field. If you are interested in working with me on anything (regarding inner alignment or anything else) please don’t hesitate to send me an email at evanjhub@gmail.com.
# The Hierarchy of Memory Reconsolidation Techniques

As discussed in the previous post, the aim is to start with the techniques that least challenge existing schema, and then work up the hierarchy to techniques that more strongly challenge the existing schema.

  

![](http://mattgoldenberg.net/wp-content/uploads/2019/11/Hierarchy-of-Reconsolidation.png)

**Experience** involves simply sitting with the schema as it is, and seeing if change comes from prolonged viewing of the schema.

**Question** involves posing questions about the schema, to draw attention to ways that it may be wrong.

**Challenge** involves directly providing counters to the schema, and seeing how it reacts.

**Dissonance** involves holding both the counter to the schema and the schema itself at once.

**Change** involves attempting to directly change how you represent the schema to take your counter into account.
# Robin Hanson on the futurist focus on AI

[![Robin Hanson](http://aiimpacts.org/wp-content/uploads/2019/11/robin_hanson.jpg)](http://aiimpacts.org/wp-content/uploads/2019/11/robin_hanson.jpg)

Robin Hanson

Robert Long and I recently talked to Robin Hanson—GMU economist, prolific [blogger](http://www.overcomingbias.com/), and longtime thinker on the future of AI—about the amount of futurist effort going into thinking about AI risk.

It was noteworthy to me that Robin thinks human-level AI is a century, perhaps multiple centuries away— much longer than the 50-year number given by AI researchers. I think these longer timelines are the source of a lot of his disagreement with the AI risk community about how much of futurist thought should be put into AI.  

Robin is particularly interested in the notion of ‘lumpiness’– how much AI is likely to be furthered by a few big improvements as opposed to a slow and steady trickle of progress. If, as Robin believes, most academic progress and AI in particular are not likely to be ‘lumpy’, he thinks we shouldn’t think things will happen without a lot of warning.

The full recording and transcript of our conversation can be found [here](https://aiimpacts.org/conversation-with-robin-hanson/).
# [Link] John Carmack working on AGI

[John Carmack](https://en.wikipedia.org/wiki/John_Carmack), confirmed GOAT video game developer, is going to take a crack at AGI.

[https://www.facebook.com/permalink.php?story_fbid=2547632585471243&id=100006735798590](https://www.facebook.com/permalink.php?story_fbid=2547632585471243&id=100006735798590)
# Books/Literature on resolving technical disagreements?

I've seen many books and schools of thought that seem to be about conflict resolution. Books like Crucial Conversations and Non-Violent Communication. There are multiple parties that want different things, there's strong emotional undertones/overtones, and these books advise you on how to navigate those conflicts and find some sort of common ground and get people what they want.

So far the Double Crux framework is the only thing I've seen that's had the explicit goal of resolving disagreements, especially disagreements about technical topics. Can anyone recommend any other books or bodies of work that have this explicit goal?
# [Personal Experiment] Counterbalancing Risk-Aversion

I am over-biased against risk. I usually take the safer option even when it's the wrong one. When I was 18 I stumbled across this paragraph.

> You must do everything that frightens you…Everything. I’m not talking about risking your life, but everything else. Think about fear, decide right now how you’re doing to deal with fear, because fear is going to be the great issue of your life, I promise you. Fear will be the fuel for all your success, and the root cause of all your failures, and the underlying dilemma in every story you tell yourself about yourself. And the only chance you’ll have against fear? Follow it. Steer by it. Don’t think of fear as the villain. Think of fear as your guide, your pathfinder… ― *The Tender Bar* by J.R. Moehringer

I followed this paragraph to the letter. Whenever I was torn between two choices and didn't know what to do I'd just take the scarier option. I recorded my results until I had accumulated 30 decisions. 28 of them (93% of the time) the scarier choice was the correct one. This immediately improved my quality of life. The two decisions where I chose wrong by doing the scarier thing were inconsequential.

I continued using fear to tiebreak my decisions for another two years. It worked great until I negated my aversion to fear. My sense of fear has become subdued outside of immediate physical danger.

I make far better decisions then I did before I deconditioned myself, but I'm still over-biased against risk. I'm not literally afraid of taking risks. I'm just overly-conservative in expected value calculations. I'm too cautious.

Having identified this problem I'm going to try something similar to what I did last time. I'm going to increase my risk tolerance slightly and record my results.

December 13, 2019 update: Apparently my last trial at this years ago wiped out most of my fears. Increasing my risk tolerance helped me write more but otherwise had little effect on my life.
# Hard to find factors messing up experiments: Examples?

In Richard Feynman's [talk](http://calteches.library.caltech.edu/3043/1/CargoCult.pdf) on cargo cults he mentions a story about experiments with rat mazes (skip if you recognize it)

> All experiments in psychology are not of this type, however.  For example, there have been many experiments running rats through all kinds of mazes, and so on—with little clear result.  But in 1937 a man named Young did a very interesting one.  He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was.  He wanted to see if he could train the rats to go in at the third door down from wherever he started them off.  No.  The rats went immediately to the door where the food had been the time before.

> The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before?  Obviously there was something about the door that was different from the other doors.  So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same.  Still the rats could tell.  Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run.  Still the rats could tell.  Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person.  So he covered the corridor, and, still the rats could tell.

> He finally found that they could tell by the way the floor sounded when they ran over it.  And he could only fix that by putting his corridor in sand.  So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door.  If he relaxed any of his conditions, the rats could tell.

I also recall some experiment where it was eventually discovered that the control of some microbiology experiment was screwed because the paper towels used in some process were made with wood that was treated with a particular chemical before it was turned into the paper towel (was this linked by gwern at some point?).

I'm interested in collecting any and all examples you have of these events; through painstaking effort someone realized that XYZ was interfering with an intended experiment (extra points if XYZ was completely outside the experimenters initial frame of "factors that might be relevant to this experiment") . Links to source if possible.
# How common is it for one entity to have a 3+ year technological lead on its nearest competitor?

I'm writing a follow-up to my [blog post on soft takeoff and DSA](https://www.lesswrong.com/posts/PKy8NuNPknenkDY74/soft-takeoff-can-still-lead-to-decisive-strategic-advantage), and I am looking for good examples of tech companies or academic research projects that are ~3+ years ahead of their nearest competitors in the technology(ies) they are focusing on.

Exception: I'm not that interested in projects that are pursuing some niche technology, such that no one else wants to compete with them. Also: I'm especially interested in examples that are analogous to AGI in some way, e.g. because they deal with present-day AI or because they have a feedback loop effect.

Even better would be someone with expertise on the area being able to answer the title question directly. Best of all would be some solid statistics on the matter. Thanks in advance!
# Do we know if spaced repetition can be used with randomized content?

Disclaimer: there may be major flaws in the way I use words. [Corrections are welcome.][crocker's rules]

---

Suppose I want to memorize all the [software design patterns].

I could use [spaced repetition] and create a new deck of flashcards. Each card would have the name of the pattern on one side and the definition on the other.

This would help me understand references to patterns without opening Wikipedia every time. This would *probably* help me recognize patterns by descriptions, as long as they're close enough to the definitions.  
But this wouldn't help me recognize patterns just by looking at their implementations. I'd have to actively think about each pattern I remember and compare the definition and the code.

I could create a second deck, with names and examples. But then I'd just memorize those specific examples and *maybe* get better at recognizing similar ones.

This problem is similar to that of testing software. (There must be a more straightforward analogy, but I couldn't find one.) Individual tests can only prevent individual errors. Formal verification is better, but not always possible. The next best thing is [fuzzing]: using random inputs and heuristics like "did it crash?".

So I wonder if I could generate new examples on the fly. (More realistically, pull hand-labeled examples from a database.)

The idea is that a skill like recognizing a pattern in the code should also be a form of memory. Or at least the parts of it that do not change between the examples. So using spaced repetition with randomized examples would be like [JIT-compilation] in brains.

There was an LW post about genetic programming working better when the environment was modular. Maybe something similar would happen here.

But I couldn't find anything on the internet. Has anybody seen any research on this?

[crocker's rules]: https://wiki.lesswrong.com/wiki/Crocker%27s_rules
[software design patterns]: https://en.wikipedia.org/wiki/Software_design_pattern
[spaced repetition]: https://en.wikipedia.org/wiki/Spaced_repetition
[fuzzing]: https://en.wikipedia.org/wiki/Fuzzing
[JIT-compilation]: https://en.wikipedia.org/wiki/Just-in-time_compilation