Harley-ml/lesswrong huggingface dataset:
# Bayes-Up: An App for Sharing Bayesian-MCQ

Inspired by [Lê Nguyên Hoang](https://www.lesswrong.com/users/le-nguyen-hoang-1)'s post on [Bayesian Examination](https://www.lesswrong.com/posts/8wzKawHmh4d3h2otw/bayesian-examination), I have been developing (as a hobby) a new app called Bayes-Up (available at: [bayes-up.web.app](https://bayes-up.web.app/)). The app is now in a state where it is working well enough to be shared with others. In this post I list a few things you can do with it, because I expect that it will spark some interest within the community.

*   **Test and improve your calibration**: Bayes-Up uses a collection of good quality trivia questions from the [open trivia database](https://opentdb.com/). The main point of the app is that you can find a list of multiple choice quizzes, answer questions by assigning probabilities to each of the possible choices, receive a score based off a quadratic proper scoring rule and later find statistics about the quality of your calibration. A good place to start is the [quiz from the book _Factfulness_ by Hans Rosling](https://bayes-up.web.app/q/VT6PgBhjXoEP) that I included in the app.
*   **Create quizzes and upload them**. There exists already a small number of calibration training apps. Bayes-Up differs mainly because it allows to upload and share your own quizzes. This can solve one of the problems of calibration apps which is to create good quality content (quizzes / questions). If you are a teacher and want your students to develop more metacognitive skills and intellectual honesty, or if you are organizing workshops on probability calibration, Bayes-Up can make it easier for you. To add a quiz, simply write it in a spreadsheet, export it as a CSV file and upload it in Bayes-Up.
*   **Recommend UI improvements, new features, report bugs, or contribute to the implementation**. Only very little feedback has been collected so far and certainly a lot could be improved with little effort. The code of the app is open source and hosted on [github](https://github.com/Stokastix/bayes-up).
*   **Analyse the data from Bayes-Up users**. So far about 30'000 questions have been answered by about 1'300 users since the end of December 2019. The collected data is available [at this link](https://github.com/Stokastix/bayes-up/blob/master/database_dump/database.csv) and will likely grow in the following months. Simple questions that analysing this data could answer are: Do users become better calibrated over time? Is calibration topic-specific or transferrable? How can the answers of users with unknown calibration and unknown knowledge be aggregated to predict the right answers to every question? Let me know if you want to do something with it or need a better documentation.
# More writeups!

A kind of post I would like to read more of is "I did X, and here's how it went." You tend to see this most with research, but I've enjoyed reading them on all sorts of things. This is one of the [main](https://www.jefftk.com/p/preparing-for-our-cd) [kinds](https://www.jefftk.com/p/boston-solstice-2019-retrospective) [of](https://www.jefftk.com/p/street-training) [post](https://www.jefftk.com/p/soundproofing-the-ceiling) [I](https://www.jefftk.com/p/folding-couch-monitor) [write](https://www.jefftk.com/p/fractional-sweets), and I would encourage others to give it a try!

Here's an arbitrary selection of writeups I've enjoyed:

*   [DB-19: Resurrecting an Obsolete Connector](https://www.bigmessowires.com/2016/06/04/db-19-resurrecting-an-obsolete-connector/)
    
*   [Your room can be as bright as the outdoors](https://www.benkuhn.net/lux)
    
*   [After one year of applying for EA jobs: It is really, really hard to get hired by an EA organisation](https://forum.effectivealtruism.org/posts/jmbP9rwXncfa32seH/after-one-year-of-applying-for-ea-jobs-it-is-really-really)
    
*   [Wood joint strength testing](https://woodgears.ca/joint_strength/)
    
*   [Giving Tuesday retrospective](https://forum.effectivealtruism.org/posts/Ns3h8rCtsTMgFZ9eH/ea-giving-tuesday-donation-matching-initiative-2018)
    
*   [Zeo sleep self-experiments](https://www.gwern.net/Zeo)
    
*   [Hexing the technical interview](https://aphyr.com/posts/341-hexing-the-technical-interview)
    
*   [EAGxBerkeley 2016 Retrospective](https://forum.effectivealtruism.org/posts/NCXKP5bv9GxmHNYx8/eagxberkeley-2016-retrospective)
    
*   [My Business Card Runs Linux](https://www.thirtythreeforty.net/posts/2019/12/my-business-card-runs-linux/)
    

Major reasons I like writeups include:

*   Exposure to problems that I didn't know were important to people.
    
*   Detailed looks at how people handled problems.
    
*   They're generally very concrete and written after the fact, which makes it harder to write something that's neat but wrong.
    
*   When I'm thinking of doing something and find a writeup by someone who did something similar it's fantastically useful.
    

When you voluntarily make something public, you do risk that people will [irresponsibly](https://www.jefftk.com/p/responsible-transparency-consumption) beat you up over failings. I think it's generally worth it to go ahead anyway, tell the whole story, and help build a norm of sharing things so others can learn.
# A Cautionary Note on Unlocking the Emotional Brain

_\[Cross-posted from [Grand Unified Crazy](https://grandunifiedcrazy.com/2020/02/08/cautionary-note-unlocking-emotional-brain/). Relevant to [Kaj's summary of the book](https://www.lesswrong.com/posts/i9xyZBS3qzA8nFXNQ/book-summary-unlocking-the-emotional-brain).\]_

In children’s stories, the good guys always win, the hero vanquishes the villain, and everyone lives happily ever after. Real life tends to be somewhat messier than this.

The world of therapy presented by [Unlocking the Emotional Brain](https://www.amazon.com/Unlocking-Emotional-Brain-Eliminating-Reconsolidation/dp/0415897173/) reads somewhat like a children’s story. Loosely, it presents a model of the brain where your problems are mostly caused by _incorrect emotional beliefs_ (bad guys). The solution to your problems is to develop or discover a _correct emotional belief_ (good guy) that contradicts your incorrect beliefs, then force your brain to recognize the contradiction at an emotional level. This causes your brain to automatically resolve the conflict and destroy the incorrect belief, so you can live happily ever after.

Real life tends to be somewhat messier than this.

After about a month of miscellaneous experimentation on myself based on this book, my experiences match the basic model presented, where many psychological problems are caused by incorrect emotional beliefs (I don’t think this part is particularly controversial in psychological circles). It also seems to be true that if I force my brain to recognize a contradiction between two emotionally relevant beliefs, it will resolve the conflict and destroy one of them. Of course, as in real life where the good guy doesn’t always win, it seems that when I do this my brain doesn’t always destroy the right belief.

I have had several experiences now where I have identified an emotional belief which analytically I believe to be false or harmful. Per _UtEB_ I have identified or created a different experience or belief that contradicts it, and smashed them together in my mind. A reasonable percentage of the time, the false belief emerges stronger than before, and I find myself twisting the previous “good” belief into some horrific experience to conform with the existing false belief.

In hindsight this shouldn’t be particularly surprising. Whatever part of your brain is used to resolve conflicting emotional beliefs and experiences, it doesn’t have special access to reality. All it has to work with are the two conflicting pieces and any other related beliefs you might have. It’s going to pick the wrong one with some regularity. As such, my recommendation for people trying this process themselves (either as individuals or as therapists) is to try and ensure that the “good” belief is noticeably stronger and more immediate than the false one before you focus on the contradiction. If this doesn’t work and you end up in a bad way, I’ve had a bit of luck “quarantining” the newly corrupted belief to prevent it from spreading to even further beliefs, at least until I can come up with an even stronger correct belief to fight it with.
# Relationship Outcomes Are Not Particularly Sensitive to Small Variations in Verbal Ability

After a friendship-ending fight, you feel an impulse to push through the pain to do an exhaustive postmortem of everything you did wrong in that last, fatal argument—you could have phrased that more eloquently, could have anticipated that objection, could have not left so much "surface area" open to that class of rhetorical counterattack, could have been more empathetic on that one point, could have chosen a more-fitting epigraph, could have taken more time to compose your reply and squeeze in another pass's worth of optimizations—as if searching for some combination of variables that would have changed the outcome, some nearby possible world where the two of you are still together.

No solution exists. (Or is findable in polynomial time.) The causal forces that brought you to this juncture are multitudinous and complex. A small change in the initial conditions only corresponds to a small change in the outcome; you can't lift a two-ton weight with ten pounds of force.

Not all friendship problems are like this. Happy endings do exist—to someone else's story in someone else's not-particularly-nearby possible world. Not for you, not here, not now.
# Who... (or what) designed this site and where did they come from?

Hi there,

I'm brand new here and I have a question for this fine community.

Who designed this site?

The reason I ask is because the design of this site is superb. Every element—everywhere I look—is visually gorgeous to look at. Not only that, but all that I find "visually gorgeous" is really well thought out.

*   The way the links preview when you hover over them (and how sometimes, they display different things _when_ you hover over them.
*   The way the comments... well, _display_ the comments. Amazing.

Holy cannoli, that comment section...

Have you seen the comment section at the bottom of the pages here? Have you seen how it works?

It's gorgeous!

I've never seen anything like that before. Someone spent some serious time figuring out that comment system.

It's a work of art!

Any information on how this site came to look so darn good _(and work so darn well)_ is much appreciated.

  

P.S. Although it wasn't the _original_ reason I joined, the "things" discussed here are my kind of "things" and, the people that discuss it, sound like they're my kind of people.

Planets seemed to have aligned.

Thanks for making this place.
# Does there exist an AGI-level parameter setting for modern DRL architectures?

Suppose the architecture includes memory (in the form of a recurrent state) and will act as the policy network for an observation-based RL agent. Evaluating the agent from a reasonable initial state, would you guess that there _exists_ a model with robustly human+ capabilities for current architectures? 

How many parameters would it take before you estimate there's a fifty-fifty chance of such a parameter setting existing? 1 billion? 1 trillion? More?
# Did AI pioneers not worry much about AI risks?

It's seems noteworthy just how little the AI pioneers from the 40s-80s seemed to care about AI risk. There is no obvious reason why a book like "Superintelligence" wasn't written in the 1950s, but for some reason that didn't happen... any thoughts on why this was the case?

I can think of three possible reasons for this:

1\. They actually DID care and published extensively about AI risk, but I'm simply not well enough schooled on the history of AI research.

2\. Deep down, people involved in early AI research knew that they were still a long long way from achieving significantly powerful AI, despite the optimistic public proclamations that were made at that time.

3\. AI risks are highly counter-intuitive and it simply required another 60 years of thinking to understand.

Anyone have any thoughts on this question?
# Will COVID-19 survivors suffer lasting disability at a high rate?

The case fatality rate of 2019-nCoV (aka Coronavirus, COVID-19) is still uncertain, with estimates floating around ranging from [0.16%-5.7%](https://www.worldometers.info/coronavirus/coronavirus-death-rate/), higher among the elderly and people with preexisting conditions, and lower among everyone else. However, the death rate doesn't capture all of the harms from infection. There is also time lost during the infection and recovery, there is the possibility of accelerated aging, and there is the possibility of long-term nonfatal disability, such as chronic fatigue.

Since 2019-nCoV has only existed for about two months, there is no data on the long-term outcomes of its survivors. However, the rate of lasting disability among survivors is important for deciding what responses are appropriate. I'm particularly interested in estimating the risk of chronic fatigue from nCoV infection. If that risk is high, this would greatly increase the importance of avoiding it personally and of suppressing it in communities of people doing important work, and would also greatly increase the expected economic impact.

As a starting point, I chose a similar but more severe virus, SARS, which was successfully contained in 2003. Out of 208 Canadian survivors of SARS, 22 (10%) appear in [this study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3071317/) of subjects "who remained unable to return to their former occupation" with "clinical similarities to patients with fibromyalgia syndrome". This implies a high lower bound on the rate of disability among SARS survivors. However, this is only one virus, and may not be representative of severe respiratory illnesses.

Good answers to this question would be:

*   Papers estimating the rates of postviral fatigue from other viruses, especially respiratory viruses, viruses with severity comparable to 2019-nCoV, and among non-elderly patients
*   Models of postviral fatigue and how they relate to 2019-nCoV
*   Data on whether and how much lung damage from non-viral sources causes chronic fatigue
*   Early data on 2019-nCoV which bears on this question

Any research help on this question is greatly appreciated, even if it provides only a bit of information about a small corner of the problem, or reports that a strategy for answering the question failed to pan out.
# How do you use face masks?

I did buy face masks a while ago, because other rationalists recommended wearing them.

The list time I had a flu and went to the doctors office I wore a face mask, the receptionist then told me to take it off while sitting in the waiting room.

While I do think I have a better idea then that receptionist, I still don't know much. It's easy to have superstitions about how to use face masks correctly. If I want to go for maximum benefit with minimum hassle, what's worth knowing? When do I throw masks away?
# Please Help Metaculus Forecast COVID-19

Metaculus has launched a new Prize. In contrast to [its earlier contest](https://www.lesswrong.com/posts/hzoYPCZfbWczbCjRB/the-bentham-prize-at-metaculus), which was focused on information to improve forecasts of relevance to Animal Welfare, [the Li Wenliang Prize](https://www.metaculus.com/questions/3667/the-li-wenliang-prize-series-for-forecasting-the-covid-19-outbreak/) is designed to incentivize such information gathering for and forecasts of COVID-19. Separate prize categories are being awarded for Forecasts, Analysis, and Question Development. [Here are the existing questions available for forecasting and analysis.](https://www.metaculus.com/questions/?search=cat:bio--infectious-disease--novel-coronavirus)
# Perceptrons Explained

This is an introduction to the perceptron learning algorithm, a proof of its convergence, and some visualizations. Ideally, in the same vein as Nicky Case's Explorable Explanations, but I don't think it's at that level yet. Only LW-tangential, mostly just an educational post into basic ML.
# Attainable Utility Preservation: Concepts

![](https://i.imgur.com/hTnYTsJ.png)


![](https://i.imgur.com/gwVocUy.png)
![](https://i.imgur.com/KPv2beS.png)

![](https://i.imgur.com/MYNBKOe.png)

![](https://i.imgur.com/ZK2qYPZ.png)

![](https://i.imgur.com/lk8Keid.png)
![](https://i.imgur.com/kMBZK6d.png)
![](https://i.imgur.com/FXlUiYj.png)
 
![](https://i.imgur.com/hHVvk0Q.png)
![](https://i.imgur.com/3NMSHHl.png)
![](https://i.imgur.com/BtzHnUq.png)

![](https://i.imgur.com/MzW64A5.png)
![](https://i.imgur.com/mOWK65o.png)

![](https://i.imgur.com/VDQiChW.png)
![](https://i.imgur.com/jtxMXJe.png)

![](https://i.imgur.com/7KcMK3J.png)

## Appendix: No free impact
What if we want the agent to single-handedly ensure the future is stable and aligned with our values? AUP probably won’t allow policies which actually accomplish this goal – one needs power to e.g. nip unaligned superintelligences in the bud. AUP aims to prevent catastrophes by stopping bad agents from gaining power to do bad things, but it symmetrically impedes otherwise-good agents. 

This doesn’t mean we can’t get useful work out of agents – there are important asymmetries provided by both the main reward function and AU landscape counterfactuals. 

First, even though we can’t specify an _aligned_ reward function, the provided reward function still gives the agent useful information about what we want. If we need paperclips, then a paperclip-AUP agent prefers policies which make some paperclips. Simple.

Second, if we don’t like what it’s beginning to do, we can shut it off (because it hasn’t gained power over us). Therefore, it has “approval incentives” which bias it towards AU landscapes in which its power hasn’t decreased too much, either. 

So we can hope to build a non-catastrophic AUP agent and get useful work out of it. We just can’t directly ask it to solve all of our problems: it doesn’t make much sense to speak of a “low-impact [singleton](https://wiki.lesswrong.com/wiki/Singleton)”.

#### Notes
* To emphasize, when I say "AUP agents do $X$" in this post, I mean that AUP agents correctly implementing the _concept of AUP_ tend to behave in a certain way. 
* As [pointed out by Daniel Filan](https://www.lesswrong.com/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure#jJrCTRwTZDZDc3XLx), AUP suggests that one might work better in groups by ensuring one's actions preserve teammates' AUs.
# Stepwise inaction and non-indexical impact measures

*Overall summary post [here](https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/PmqQKBmt2phMT7YLG).*


In [a previous post](https://www.lesswrong.com/posts/fqJmZBG5xmvvd8WRi/impact-measures-and-subagents), I asked which impact measures were vulnerable to subagents. Vika [pointed out](https://www.lesswrong.com/posts/fqJmZBG5xmvvd8WRi/impact-measures-and-subagents#uuyjSqyuXYnHg4dkY) that it was not merely an issue of of the impact measure, but also of the baseline.

This is indeed the case, but the nature of the impact measure is still relevant. In this post, I'll establish two facts: that under the stepwise inaction baseline, a subagent completely undermines all impact measures (including [twenty billion questions](https://arxiv.org/pdf/1705.10720.pdf)). And for the inaction baseline, for non-indexical impact measures, a subagent will not change anything.

The next post will delve into the very interesting things that happen with the inaction baseline and an indexical impact measure.

# Stepwise inaction baseline

All impact measures are some form of distance measure between two states, of the form $\rho(s_t,s_t')$, where $s_t$ is the state the agent is actually in, and $s_t'$ is some baseline state to compare with.

For the stepwise inaction baseline, $s_t'$ is calculated as follows. Let $s_{t-1}$ be the previous state, $a_{t-1}$ the previous action. Then $s_t'$ is the state that would have followed $s_{t-1}$ had the agent taken the noop action, $\varnothing$, instead of $a_{t-1}$.

However, in the presence of a subagent, the agent merely has to always take the action $a_{t-1}=\varnothing$. In that case, $s_t=s_t'$ (in a deterministic setting[^nondet]), and the penalty $\rho(s_t,s_t')$ collapses to nothing.

[^nondet]: In a non-deterministic setting, $\rho(s_t,s_t')$ becomes a mix of an impact penalty and a measure of environment stochasticity.

This leaves the subagent free to maximise $R_0$ (the positive reward for agent and subagent), without any restrictions beyond making sure that the agent can always take the $\varnothing$ action.

# Non-indexical impact penalty

The  [twenty billion questions](https://arxiv.org/pdf/1705.10720.pdf) is an example of a non-indexical impact penalty. The impact penalty is defined as the difference in expected value of many variables, between the current state and the baseline state.

With the inaction baseline state, $s_t'$ is the state the system would have been in, had the agent always done nothing/had not been turned on. Thus the definition of $s_t'$ is independent of the agent's actions, so the inaction baseline does not collapse like the stepwise inaction baseline does.

What about subagents? Well, since the impact penalty is non-indexical, a subagent does not get around it. It matters not whether the subagent is independent, or an extension of the agent: the impact penalty remains.
# Subagents and impact measures: summary tables

These tables will summarise the results of this whole sequence, checking whether subagents can neutralise the impact penalty.

First of all, given a subagent, here are the results for various impact penalties and baselines, and various "value difference summary functions" $f$:

$$\begin{array}{|c|c|c|c|}
\hline
\textrm{Impact Penalty}&& \begin{array}{c}\textrm{Penalty neutralised} \\ \textrm{- sw inaction?}\end{array} & \begin{array}{c}\textrm{Penalty neutralised} \\ \textrm{- inaction?}\end{array} \\
\hline\hline \textrm{Non-indexical} && \textrm{Yes} & \textrm{No} \\
\hline \textrm{Indexical, $f$ decreasing} && \textrm{Yes} & \textrm{No}\\
\hline \textrm{Indexical, $f$ increasing} && \textrm{Yes} & \textrm{Mostly}\\
\hline
\end{array}$$

Another way of phrasing "$f$ decreasing": it penalises too little power, not too much. Conversely, "$f$ increasing" penalises too much power, not too little. Thus, unfortunately:

* Subagents **do allow** an agent to get **stronger** than the indexical impact penalty would allow.
* Subagents **don't allow** an agent to get **weaker** than the indexical impact penalty would allow.


## Examples

This table presents, for three specific examples, whether they could actually build a subagent, and whether that would neutralise their impact penalty in practice (in the inaction baseline):

$$\begin{array}{|c|c|c|c|}
\hline
\textrm{System}&& SA\textrm{ possible?} & \textrm{Penalty neutralised?} \\
\hline\hline \textrm{20BQ} && \textrm{Yes} & \textrm{No} \\
\hline \textrm{RR} && \textrm{Yes} & \textrm{No}\\
\hline \textrm{AU} && \textrm{Probably} & \textrm{Mostly}\\
\hline
\end{array}$$

Here, 20BQ is [twenty billion questions](https://arxiv.org/abs/1705.10720), RR is [relative reachability](https://arxiv.org/pdf/1806.01186.pdf), and AU is [attainable utility](https://arxiv.org/abs/1902.09725).

Now, whether the RR or AU penalties are undermined technically depends on $f$, not on what measure is being used for value. However, I feel that the results undermine the spirit of AU much more than the spirit of RR. AU attempted to control an agent by limiting its power; this effect is mainly neutralised. RR attempted to control the side-effects of an agent by ensuring it had enough power to reach a lot of states; this effect is not neutralised by a subagent.
# [Productivity] How not to use "Important // Not Urgent"

_Epistemic status: Personal experience._

This is an **Eisenhower matrix**, named after President Dwight Eisenhower.

![](https://i.pinimg.com/474x/6a/a7/01/6aa7010c0a37a89c64a44145da4e0f86.jpg)

Eisenhower matrices have a very useful, _very specific_ role in my life these days.

I usually bust one out on paper when I'm feeling overwhelmed by my own anxiety, and use it to focus my attention on what's in the first quadrant. I tell myself that I'll return to it by the time I finish the first quadrant, but usually by the time I've actually filled that quadrant out, my panic has subsided completely and I feel no issues in discarding the 2x2.

For a while, I also used the Eisenhower matrix as [the basis for my Todoist Premium workflow](https://medium.com/@aquinn.usa/how-i-currently-use-todoist-premium-b92f0f10e771). I actually still think that's a pretty good system for someone just getting started with using Todoist, and wanting a little bit more structure with how to use it outside of the box.

But I noticed that often, I would add things that I wanted to do way out in the future to the "Important // Not Urgent" filter. This is a bad idea, because economics tells us that ideas of things you want to do in the far future should probably be [discounted](https://www.investopedia.com/terms/d/discounting.asp) quite heavily, relative to things you want to do in maybe the next month or so. You might well find that 5 years from now, you don't give a shit about learning to play Vivaldi or looking into moving to a different place; those are all things you can leave as amorphous ideas to play around with in your mind.

So my advice: **Don't use the Urgent // Not Important quadrant for anything that isn't due more than two weeks out.** That's a pretty aggressive cutoff point, yes, but we're talking planning your personal life here, not business; and we're talking a tool meant to focus your executive function in good directions. You don't want to overstretch it.

One objection that I could forsee someone saying is, "Well, what if I have a recurring task that I want to do every day, but it doesn't pay off immediately for me?", to which I say, the todo list is usually a fundamentally bad format for that, because you're probably talking about something that resembles practice (exercise, anything artistic, even studying).

Practice routines are [time-delimited activites, not task-delimited ones](https://www.lesswrong.com/posts/Foi6mmbkWj6QBLWsP/productivity-task-vs-time-delimitation). You usually want to do them for similar intervals of time, and they are usually done inside of similar environments. Use a calendar instead. Even if you have to go in every day and tweak the exact time you plan to do it, it's a much more natural fit than a todo list, which shines brightest for the thousand one-off things you have to do each day.
# Counterfactuals versus the laws of physics

*Overall summary post [here](https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/PmqQKBmt2phMT7YLG).*

I've [claimed](https://www.lesswrong.com/posts/JB6edzY5cccrdbQxP/stepwise-inaction-and-non-indexical-impact-measures#xmJBmgCpdYkgrQTS7) that, given a subagent:

>[...] restrictions on increased power for the agent ("make sure you never have the power to increase the rewards") become restrictions on the actual policy followed for the subagent ("make sure you never increase these rewards").

But how does this work? The problem is that "never have the power" is a counterfactual statement: it doesn't matter what you do, only what you could potentially do. But "never increase" is the opposite: it matters not what you could potentially do, only what you do.

How does the subagent change this? Well, the counterfactual version is at least informally clear: the agent $A$ could, by changing its policy, increase certain rewards.

But now suppose that $A$ creates a subagent $SA$ that will itself not increase the rewards, and, moreover, will interfere with $A$ if $A$ attempts to do so. Now $A$ cannot increase those rewards *even if it changed its policy*. The laws of the environment - which, from $A$'s perspective, now include $SA$ - prevent $A$ from ever being able to to increase the rewards, and thus fulfil $A$'s penalty function.

This is very much the $A$ "[chaining itself to the mast](https://en.wikipedia.org/wiki/Ulysses_pact)".

![](https://www.dropbox.com/s/ldy54kg7amdj1yq/sirens.jpg?raw=1)

## The counterfactual we'd want (but don't have)

What we want is for the counterfactual of the impact penalty to also include the subagent: the agent and subagent together definitely have the "power" to change the rewards by changing their own policies. But, from $A$'s perspective, $SA$'s policy is a brute fact about the world, not amenable to change.

We'd like for the counterfactual to also include $SA$'s policy, but for that, we'd need to identify and define subagents - a [very tricky problem](https://www.lesswrong.com/posts/8RCCMStERhfkYZC8i/the-subagent-problem-is-really-hard).
# Big Yellow Tractor (Filk)

_To the tune of "Big Yellow Taxi", with apologies to Joni Mitchell_

we'll pave paradise and put up a parking lot  
if that's what it takes to make all the wild suffering stop  
the pain it only grows and grows  
until we boil it up in our pot  
we'll pave paradise and put up a parking lot  

we'll take all the creatures and put 'em in a simulation  
where there's no pain or death or starvation  
oh! the pain it only grows and grows  
until we boil it up in our pot  
we'll pave paradise and put up a parking lot  

hey all you people put away decision theory  
i don't care about max utils if we can't end suffering, please!  
the pain it only grows and grows  
until we boil it up in our pot  
we'll pave paradise and put up a parking lot  

late last night I heard 'bout a brand new plan  
a big yellow tractor's gonna clear away all the land  
oh! the pain it only grows and grows  
until we boil it up in our pot  
we'll pave paradise and put up a parking lot
# Blog Post Day (Unofficial)

_TL;DR: You are invited to join us [online](https://discord.gg/j5TzpVF) on Saturday the 29th, to write that blog post you've been thinking about writing but never got around to. Comment if you accept this invitation, so I can gauge interest._

The Problem:
------------

Like me, you are too scared and/or lazy to write up this idea you've had. What if it's not good? I started a draft but... Etc.

The Solution:
-------------

**1\. Higher motivation via Time Crunch and Peer Encouragement**

We'll set an official goal of having the post put up by midnight. Also, we'll meet up in a [special-purpose discord channel](https://discord.gg/j5TzpVF) to chat, encourage each other, swap half-finished drafts, etc. If like me you are intending to write the thing one day eventually, well, here's a reason to make that day this day.

**2\. Lower standards via Time Crunch and Safety in Numbers**

Since we have to be done by midnight, we'll all be under time pressure and any errors or imperfections in the posts will be forgivable. Besides, they can always be fixed later via edits. Meanwhile, since a bunch of us will be posting on the same day, writing a sloppy post just means it won't be read much, since everyone will be talking about the handful of posts that turn out to be really good. If you are like me, these thoughts are comforting and encouraging.

Evidence this Works:
--------------------

MIRI Summer Fellows Program had a Blog Post Day towards the end, and it was enormously successful. It worked for me, for example: It squeezed two good posts out of me. (OK, so one of them I finished up early the next morning, so I guess it technically doesn't count. But in spirit it does: It wouldn't have happened at all without Blog Post Day.) More importantly, MSFP keeps doing this every year, even though opportunity cost for them is much higher (probably) than the opportunity cost for you or me. I don't know what else you had planned for Saturday the 29th... (Actually, if you do have something else planned, but otherwise want to participate in Blog Post Day, let me know. Maybe we can pick a different day.)

Side Benefits:
--------------

It'll be fun!
# Is there software for goal factoring?

Is there software for goal factoring (the CFAR technique)? I want to use it to create a directed graph, which is not necessarily a tree and may have cycles (this requirement disqualifies mind mapping software). Nodes are goals, they have attached text. There's an edge from goal x to goal y iff fulfillment of x directly contributes to fulfillment of y. There should be an easy way to see the graph and modify it, preferrably in a visual way.
# We Want MoR (HPMOR Discussion Podcast) Completes Book One

The _We Want MoR_ podcast, a chapter by chapter read-through and discussion of _Harry Potter and the Methods of Rationality_ by a Rationalist and a newbie, has just finished up [Book One](https://www.doofmedia.com/2020/02/18/we-want-mor-book-one-retrospective-ch-1-21/) (Ch. 1-21) of HPMOR. ([Apple Podcasts link](https://podcasts.apple.com/us/podcast/we-want-mor-chapter-18/id431784580?i=1000461672191).)

Less Wrong used to be a major locus of discussion for HPMOR, so I thought it made sense to share this here, especially now that the show has passed this milestone. If folks don't necessarily want to keep HPMOR content on Less Wrong, there are ongoing lively discussions of the weekly episodes on the [r/hpmor](https://www.reddit.com/r/hpmor) subreddit.

Disclosure, I was the guest on this episode.
# Is there an intuitive way to explain how much better superforecasters are than regular forecasters?

Is there an intuitive way to explain how much better superforecasters are than regular forecasters? (I can look at the tables in [https://www.researchgate.net/publication/277087515\_Identifying\_and\_Cultivating\_Superforecasters\_as\_a\_Method\_of\_Improving\_Probabilistic_Predictions](https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions) but I don't have an intuitive understanding of what brier scores mean, so I'm not sure what to think about it).
# Stuck Exploration

Recently, we've been considering agents that instead of just being given the problem setup have to actually learn it from experience. For example, instead of being told that an agent is a perfect predictor, they just have to play until they realise this. When an agent encounters a problem like iterated [Parfit's Hitchhiker](https://wiki.lesswrong.com/wiki/Parfit's_hitchhiker), there is a weird effect that can occur during exploration if the predictor is perfect.

We will turn Parfit's Hitchiker into an iterated problem by changing being left in the desert to a large utility penalty, rather than just dying. Suppose an agent has decided that next time it is in town that it won't pay. As long as it is committed to this, it'll never actually end up in town and so it'll never actually fulfil this commitment. It'll always be predicted to defect and so will never get the chance. We will call this stuck exploration because the exploration never resolves.

One trivial way around this is to never commit based on the next time, but instead control exploration by a pseudo-random variable based on the time. However, avoiding _Stuck Exploration_ doesn't necessarily mean that the agent will necessarily end up understanding the situation. The agent will notice that it always pays in town; ie. that it never takes the explore option when there. An EDT agent would be able to figure out that it is likely being predicted, while a dualistic agent like AIXI wouldn't be able to directly make this connection. Of course, AIXI would notice that something weird was happening and this would distort the model of the algorithm it chooses in some way. Perhaps it would a spurious link, but we won't try to delve into this at this stage.

_This post was a result of discussions with Davide Zagami and supported by the EA Hotel and AI Safety Research Program. Thanks to Pablo Moreno and Luke Miles for feedback._
# Making Sense of Coronavirus Stats

_\[Update: Some have pointed out the definition of mortality rate should be that of deaths to some defined population, typically a median estimate over a period of time (week, month, year...) and not limited only to those infected. (New update here. The ratios I have been considering are called_ Case Fatality Rate.)

_It was also correctly pointed out that my second calculation was simply wrong. The ration should not be deaths/recovered but deaths/(deaths+recovered) -- that is deaths to the total population considered. \]_

From what I've seen WHO and other health organizations are saying the mortality rate for the new coronavirus outbreak is between 2 and 3 percent. That seems to be based on the ratio of the reported deaths to the reported cases of infection.

That doesn't seem right to me.

The last statistics I looked at (news report) was:

Total - 75,768

Recovered - 16,329 (21.6%)

Deaths - 2129 (2.8%)

However, that leave us with a bit over 75% with an unknown end state.

If I try to infer the outcome for all the reported infections I can think of two ways to estimate the end results. One, is to assuming the remaining cases will produce a similar outcome as has been observed so far. Using that assumption I can then iterate through the unknown cases using the 21.6% percent will recover, 2.8% will die and ~75% will move to the next round.

The other way would be to assume the ratio of the current deaths to current recoveries is a good measure of the mortality rate.

Using the first approach the total deaths approaches 8,740 people from the current population of 75,768. The second approach results in a higher number, 9879. Both of these numbers would suggest this version of the coronavirus is pretty bad, in terms of mortality rates. In the context of the other two coronavirus outbreaks, it seem closer to SARS, though a bit worse, than to MERS.

Should I think this approach to estimating mortality rates for new diseases without out a long history (like the flu) might be more accurate than the standard approach that seems to be deaths/total infections. This standard approach would seem to systematically under estimate the mortality as initially one would expect infections to be rising more rapidly than deaths.

The implication is that health organizations/bureaucracies might tend to be slow to react. The early numbers are nothing to worry about, and might even show a decreasing mortality rate initially. Then as the infected either succumb to the infection or recover mortality rates start rising producing greater concern and calls for action.

That seems to reflect the general response both from China and from WHO in many ways.

So as I've been writing this I've come to wonder if the way mortality rates are calculated might lead to poor bureaucratic responses. I wonder if we used either of the above, rather than the standard measure, might not be better. At the end of the day, both measures will converge to the same number over time as the daily, weekly, monthly or even annual data points become too small to really move the dial at all.
# How do you survive in the humanities?

Two years ago, I fulfilled a huge part of my dream of doing what I love and started studying creative writing full time. This program is run by a humanities department, so I knew I'd have to stomach epistemic nonsense from time to time, which most of the time I can do while reminding myself that in the end it's the diploma and the connections that I care about.

But some days are harder than others, and every time a teacher says something plainly wrong, I can't seem to decide between the cost of derailing the class and the cost of letting my classmates walk home with a wrong idea. For example, in first semester I had a long discussion with a teacher about questions in general; he didn't believe that questions are supposed to get answered, that debates are supposed to be settled. I tried my best to make the case that a long-standing debate managed in such a way that it never ends is a very poorly managed debate, but he thought that was the whole point of a debate. This was the same teacher who later said that an essay could make strong points with impeccable arguments and solid evidence and still he had the essential freedom to not believe it. Last year, a small disagreement over the interpretation of a poem got seriously derailed to the definition of life, and the teacher started saying one badly constructed modus ponens after another, and I didn't know whether to address each argument or to simply point out he didn't know how to make an argument and should never have been given a teaching position. Last week, a teacher recited the standard clichés against logical positivism and claimed that science was oblivious to the suffering of the poor; upon further probing, it turned out he was confusing positivism with optimism. I wasted half an hour trying to explain his error, to no avail. At one point he uttered the phrase "I prefer to believe..." and it was impossible to make him see how irresponsible such a statement is. Yesterday, a teacher was explaining the composition of a literary essay, and she claimed that an essay writer isn't required to provide justification for their claims. I asked, "Then why should I believe anything the essay says?" and she replied, "You're free to decide whether you believe it or not," and I was just too exhausted from last week to explain that's not how beliefs should work. She then proceeded to teach social constructionism while my insides leaked plutonium and melted down.

I knew from the beginning what I was getting into. But it's hard not to despair for the current state of education in the world. Art departments live in a separate universe and they teach generation after generation an epistemically irresponsible view of the world. Do you think it's worth my energy to try to correct my teachers? Most of my classmates already agree with the view that objective facts don't exist and rationality hinders your creativity, and every day I feel like an infiltrated agent in enemy territory. Should I just shut up and focus on graduating? Or would it be unethical of me to just stand by while hundreds are taught to shut off their reasoning skills?
# Curiosity Killed the Cat and the Asymptotically Optimal Agent

[Here's](https://e563b909-928d-4538-97f1-e473938f7515.filesusr.com/ugd/421795_b6f900ffdc194b6b88c25bbb48fb5b63.pdf) a new(ish) paper that I worked on with Marcus Hutter.

If an agent is exploring enough to be guaranteed strong performance in the limit, that much exploration is enough to kill it (if the environment is minimally difficult and dangerous). It's nothing too surprising, but if you're making a claim along these lines about exploration being dangerous, and you need something to cite, this might work.

My attitude towards safe exploration is: exploration isn't safe. Don't do it. Have a person or some trusted entity do it for you. The paper can also be read as a justification of that view.

Obviously, there are many more details in the paper.
# Northwest Passage Update

Singing [Stan Rogers](https://en.wikipedia.org/wiki/Stan_Rogers)' [Northwest Passage](https://en.wikipedia.org/wiki/Northwest_Passage_(song)) with friends this evening I realized these days it needs an update. People suggested lyrics, and here's a filked tag to sing after the song:

> *In this warm clime, I have sailed the Northwest Passage  
> Opened by our fires, of coal and industry  
> Tracing one warm line through a land no longer frozen  
> Until we cool our planet down again  
> Sequester all this carbon once again*

Lyrics by Kip, Mel, Lisa, me, and others.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100143963792242)*
# The recent NeurIPS call for papers requires authors to include a statement about the potential broader impact of their work

[NeurIPS](https://en.wikipedia.org/wiki/Conference_on_Neural_Information_Processing_Systems) (formerly NIPS) is a top conference in machine learning and computational neuroscience. The recently published [call for papers](https://neurips.cc/Conferences/2020/CallForPapers) for NeurIPS 2020 includes the following (which did not appear in previous years):

>In order to provide a balanced perspective, authors are required to include a statement of the potential broader impact of their work, including its ethical aspects and future societal consequences. Authors should take care to discuss both positive and negative outcomes.
>
>[...]
>
>**Evaluation criteria:** [...] Submissions will also be considered on ethical grounds. Regardless of scientific quality or contribution, a submission may be rejected for ethical considerations, including methods, applications, or data that create or reinforce unfair bias or that have a primary purpose of harm or injury.
# Other versions of "No free lunch in value learning"

Here are some more general results derived from "[Occam's razor is insufficient to infer the preferences of irrational agents](https://arxiv.org/abs/1712.05812)":

* Regularisation is insufficient to make inverse reinforcement learning work.
* Unsupervised learning cannot deduce human preferences; at the very least, you need semi-supervised learning.
* Human theory of mind cannot be deduced merely be observing humans.
* When programmers "correct an error/bug" in a value-learning system, they are often injecting their own preferences into it.
* The implicit and explicit assumptions made for a value-learning system, determine what values the system will learn.
* No simple definition can distinguish a bias from a preference, unless it connects with human judgement.

These are all true; the key question is to what extent they are true. Do we need to have minimal supervision, or add minimal assumptions, to get the AI to deduce our values correctly? After all, we do produce a lot of data, that it could use to learn our values, if it gets the basic assumptions right.

Or do we need to put in a lot more work in the assumptions? After all, most of the data we produce is by humans, for humans, who will accept the implicit part of the data, hence there are few explicit labels for the AI to use.

My feeling is that we probably only need a few assumptions - but maybe more than some optimists in this area believe.