Harley-ml/lesswrong huggingface dataset:
# Gauging the conscious experience of LessWrong

Yesterday I discovered I had been a victim of the Typical Mind Fallacy thanks to accidentally finding out that someone very close to me doesn’t have an internal monologue, doesn’t think in words and has a visual photographic memory. Despite knowing that such people existed I thought I would know if someone close to me was one. As a result I thought it would be an interesting and fun exercise to gauge the conscious perceptions of the community. 

The below questions are judged on a 0-99 scale (they are not binary or predictions, the embedded Elicit formatting was just too great to not use), they attempt to find where the reader sits on a series of common mental metrics. 

How vivid is your visual imagination?

Elicit Prediction ([elicit.org/binary/questions/VMs-r172N](elicit.org/binary/questions/VMs-r172N))

How vivid is your sound imagination?

Elicit Prediction ([elicit.org/binary/questions/7IdTsZLwQ](elicit.org/binary/questions/7IdTsZLwQ))

How vivid is your taste imagination?

Elicit Prediction ([elicit.org/binary/questions/1z38N4iwb](elicit.org/binary/questions/1z38N4iwb))

How vivid is your smell imagination?

Elicit Prediction ([elicit.org/binary/questions/LXJq-IvZ7](elicit.org/binary/questions/LXJq-IvZ7))

How vivid is your touch imagination?

Elicit Prediction ([elicit.org/binary/questions/DYD1BjiN0](elicit.org/binary/questions/DYD1BjiN0))

Do you have an internal monologue?

Elicit Prediction ([elicit.org/binary/questions/0klHZhAEf](elicit.org/binary/questions/0klHZhAEf))

How frequently do you think in words?

Elicit Prediction ([elicit.org/binary/questions/u2R0BtAwp](elicit.org/binary/questions/u2R0BtAwp))

How good is your memory?

Elicit Prediction ([elicit.org/binary/questions/dyZ0gw67E](elicit.org/binary/questions/dyZ0gw67E))

How much control do you have over your mind? (For example, the ability to stop all thoughts on command)

Elicit Prediction ([elicit.org/binary/questions/eULBN11TJ](elicit.org/binary/questions/eULBN11TJ))

Do you have a type of Synaesthesia? (0 = none, 100 = very intense and encompassing)

Elicit Prediction ([elicit.org/binary/questions/cc-L-MZgK](elicit.org/binary/questions/cc-L-MZgK))

If anybody has an unusual mental experience (or normal one!) please share it with us in the comments. If you do, don’t leave out the obvious, try to avoid the illusion of transparency.
# How Hard Would It Be To Make A COVID Vaccine For Oneself?

I always assumed, without investigating, that vaccine-making was the sort of thing which required highly specialized experts and equipment. But I've been hearing that the Moderna vaccine was designed in two days, which strongly suggests that it's *simple*. So I'm at least asking the question: how hard would it be to make such a vaccine?

The main question is what's involved in designing and synthesizing a vaccine. Subquestions:

*   What information is needed for design, and is any non-public?
*   What skills are needed for design (other than general bio/bioinformatics know-how), and how does one design a vaccine?
*   What equipment and supplies are needed for synthesis?
*   Does the equipment require any unusual skills (other than typical wetlab stuff, e.g. how to use a pipette or prevent contamination)?
*   Perhaps most important for a non-professional effort: what quality assurance steps are typically used? How can one verify that the thing one intended to produce was actually produced?

These are the main things I'd like to know about.

There's also a cluster of secondary questions around exactly what one could do with such a vaccine, beyond using it oneself. Presumably the FDA would shut down any effort to sell it to others as a vaccine, but which part of that is the problematic part? If someone gives away a homebrew vaccine for free, is that ok? If it's advertised as not-a-vaccine (but with a technical explanation of exactly what it is), or even as "not for human use" (since of course humans never use things not fit for human use), is that ok? To what extent can barriers be circumvented by loading all the equipment in an RV and taking a weekend trip to Canada/Mexico/whatever the equivalent is for Europe? I'm imagining e.g. a small group in Berkeley decides to put in the effort to learn how to make vaccines, buys $10k of equipment and supplies, then gives away vaccine to most of the Berkeley rationalist hub (with a clear "not approved by FDA, not fit for human consumption" warning accompanied by an implicit wink). Where does the FDA draw the line between "a group of friends doing stuff they shouldn't be doing" vs something which needs to be shut down?

I expect any viable plan would take quite a lot of effort, but there's an awful lot of value to be had here - even just immunizing oneself and a handful of friends could plausibly be valuable enough to justify the effort and expenditure. Also, this seems like the sort of thing where someone could learn the skills and acquire the equipment in advance, as preparation for future pandemics - possibly including diseases much more dangerous than COVID. Given how well this episode has gone, that could provide even more value.
# Babble Challenge: Not-So-Future Coordination Tech

In contrast to [John's question](https://www.lesswrong.com/posts/w4kbvQjPy6HEKFNdA/what-would-advanced-social-technology-look-like), I'm looking for things which could easily happen if only we *had the idea,* facilitated by technology today, or (even better) with nothing but pen and paper, raising hands, etc.

Think about the difference between small group coordination *without* the idea of taking a vote, vs *with.* That's the sort of difference I'm after.

Or driving with/without the idea of staying on the right side of the road.

Or a world with/out kickstarter.

If you want a numerical mark, come up with 20. Ideas can obviously be super vague (EG "like kickstarter, but for Effective Altruism"), since I'm not expecting you to EG *actually* come up with a better voting system or a way to make futarchic governance practical for small groups, etc.

Feel free/encouraged to list "obvious" ideas, including ideas already mentioned by other people, to keep the creative juices flowing.
# Second Solstice** (Europe)

***still not actually Solstice*

We'll be doing a second running of the online Solstice ceremony this Saturday, this time more European-Timezone-Friendly.

![](https://scontent.xx.fbcdn.net/v/t1.15752-0/p206x206/132576128_236830507796299_929464869341990341_n.jpg?_nc_cat=105&ccb=2&_nc_sid=58c789&_nc_ohc=HAAeQZOz9zIAX85ELot&_nc_ad=z-m&_nc_cid=0&_nc_ht=scontent.xx&tp=6&oh=c6c770cd4853c0ef6a32123a0ef227e2&oe=60078392)

You can RSVP on the [facebook event here](https://www.facebook.com/events/146467817261217/).
# Moral moralizing

Most people get their ethics from a combination of trusting what is normally done in their society and doing what they feel is right.

It seems to me that this has been utterly discredited as a reliable source of ethical advice, because it is the same one that recommended to the average person slavery as labor, and genocide as standard geopolitical strategy, and rape as prize, and torture as entertainment.

I don’t know of a clearly better source, for a typical person who isn’t a professional ethicist. But given the intense fallibility of this one, I’m inclined to say that the resulting moral views should be held with uncertainty and modesty.

From this I’m inclined to infer that people should not be _moralizing_ about moral disagreements.

I do share the urge to moralize when I’m in a moral disagreement, but is this wrong?
# Announcing AXRP, the AI X-risk Research Podcast

Happy holidays! Today, I’m launching AXRP (pronounced axe-urp), the AI X-risk Research Podcast. The episodes involve me interviewing a researcher about a paper they’ve written, talking about the ideas in the paper and why they matter. The first three guests are [Adam Gleave](https://www.gleave.me/), [Rohin Shah](https://rohinshah.com/), and [Andrew Critch](http://acritch.com/). You can listen on all major podcast distribution networks by searching “AXRP”, or read transcripts online at [axrp.net](https://axrp.net/). If you have comments about the show, feel free to leave them here, or to email me at [feedback@axrp.net](mailto:feedback@axrp.net).
# New Bucket Brigade UI

After using the UI that the [Ritual Engine](https://github.com/dspeyer/ritualEngine) folks built for [Solstice](https://www.jefftk.com/p/secular-solstice-2020), I was feeling a bit embarrassed with how awkward and confusing stand-alone Bucket Brigade was:

[![](https://www.jefftk.com/bucket-brigade-old-ui.png)](https://www.jefftk.com/bucket-brigade-old-ui-big.png)

I took a pass over it to make the UI more coherent:

[![](https://www.jefftk.com/bucket-brigade-new-ui.png)](https://www.jefftk.com/bucket-brigade-new-ui-big.png)

[![](https://www.jefftk.com/bucket-brigade-new-ui-buckets.png)](https://www.jefftk.com/bucket-brigade-new-ui-buckets-big.png)

There are a lot of changes, but the biggest ones:

*   There is now an explicit concept of 'bucket' in the UI:
    
    *   Names are drawn in the appropriate bucket for their position.
        
    *   You can join a bucket by clicking a button.
        
    *   You shouldn't have to deal with numeric positions.
        
    *   If you do need it there is still a box where you can manually enter a position, and positions are displayed in the mixing console
        
*   It uses the whole page instead of making you scroll around:
    
    *   Links are across the top
        
    *   Controls are across the bottom
        
    *   If there is too much to display in the main panel, just not panel scrolls.
        
*   There are multiple main panels, with a tabbed interface.
    
*   Settings for configuring roundsinging are out of the main flow, on the "Advanced Settings" tab.
    
*   Debug settings are divided into ones that only affect you versus ones that affect everyone.
    
*   There's a setting for adjusting how loud the backing track / metronome is, if you're using one.
    
*   Instead of looking at the user agent string to determine whether the browser supports what we need, we specifically test for them (`if (window.AudioWorklet) ...`).
    
*   It is a (debatably) tasteful blue, and has a logo.
    

Since the app only works in modern browsers anyway, I styled the UI with [flexbox](https://css-tricks.com/snippets/css/a-guide-to-flexbox/). It's wonderful that there is now a way to use CSS that is solidly better than tables.

I'm pretty happy with it, and while we still don't have automated tests it seems to work correctly with my poking at it. If you find any problems or have suggestions, please let me know either in the comments here or by [filing an issue](https://github.com/gwillen/solstice-audio-test/issues/new).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100198390570542)*
# MIRI: 2020 Updates and Strategy

Highlights from MIRI's year-end strategy update:

*   MIRI executive director Nate Soares reports that the "[new research directions](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/)" we initiated [in 2017](https://intelligence.org/2017/04/30/2017-updates-and-strategy/) have "at this point, largely failed, in the sense that neither Eliezer nor I have sufficient hope in it for us to continue focusing our main efforts there. \[...\] We are currently in a state of regrouping, weighing our options, and searching for plans that we believe may yet have a shot at working."
*   Our most interesting (to us) research we've written up publicly this year is Scott Garrabrant's [Cartesian Frames](https://www.lesswrong.com/posts/BSpdshJWGAW6TuNzZ/introduction-to-cartesian-frames) and Vanessa Kosoy and Alex Appel's [Infra-Bayesianism](https://www.lesswrong.com/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence).
*   Our funding situation is good, and we aren't running a fundraiser this winter.
*   We're (still) seriously thinking about the pros and cons of [moving out of the Bay Area](https://www.lesswrong.com/posts/FghubkDy6Dp6mnxk7/the-rationalist-community-s-location-problem?commentId=WggEETNGjd6d2XRgR#MPM6gaXJiCnGgEmDJ), and thinking about what other locations have to offer.

You can find the full post [here](https://intelligence.org/2020/12/21/2020-updates-and-strategy/).
# The First Sample Gives the Most Information

*I originally heard this point made by Ben Pace in Episode 126 of* [*the Bayesian Conspiracy Podcast*](http://www.thebayesianconspiracy.com/2020/12/126-lesswrong-2-0-and-books/)*. Ben claimed that he learned this from the book* [*How to Measure Anything*](https://hubbardresearch.com/publications/how-to-measure-anything-book/)*, but I think I identified the relevant section, and this point wasn't made explicitly.*

Suppose that I came up to you and asked you for a 90% confidence interval for the weight of a wazlot. I'm guessing you would not really know where to start. However, suppose that I randomly sampled a wazlot and told you it weighed 142 grams. I'm guessing you would now have a much better idea of your 90% confidence interval (although you still wouldn't have *that* good a guess at the width). 

In general, if you are very ignorant about something, the first instance of that thing will tell you what domain you're operating in. If you have no idea how much something weighs, knowing the weight tells you the reasonable orders of magnitude are. Things that sometimes weigh 142 grams don't typically also sometimes weigh 12 solar masses. Similarly, things that take 5 minutes don't typically also take 5 days, and things that are 5 cm long aren't typically also 5 km long.

For more abstract concepts, having a single sample allows you to locate the concept in [concept space](https://www.lesswrong.com/posts/82eMd5KLiJ5Z6rTrr/superexponential-conceptspace-and-simple-words) by anchoring it to [thing space](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace). "Redness" cannot be properly understood until it is known that "apples are red". "Functions" are incomprehensible until you know "adding one to a number" is a function. "Resources" are vague until you learn that "money is a resource".

In reality, the first sample often gives you *more* information than a random sample. If I ask a friend for an example of a snack, they're not going to randomly sample a snack and tell me about it; they're probably going to pick a snack that is at the center of the space of all snacks, like potato chips. 

From an information-theoretic perspective, the expected amount of information gained from the first sample must be the highest. If the sampling process is independently and identically distributed, the 2nd sample is expected to be more predictable given knowledge of the first sample. There is some chance that the first sample is misleading, but the probability that it's misleading goes down the more misleading the sample is, so you don't *expect* the first sample to be misleading. If you're very ignorant, your best guess for the mean of a distribution is pretty close to the mean of the samples you have, even if you only have one.

This is one perspective on why asking for examples is so powerful; they typically give you the first sample, which contains the most information.
# The medical test paradox: Can redesigning Bayes rule help?

A great coverage of one of my favorite Youtuber's of the (on LessWrong) classical reformulation of Bayes rule in terms of odds instead of probabilities. I expect to link a bunch of people to this, in particular if they get stuck reading [Eliezer's intuitive explanation of Bayes](https://www.lesswrong.com/posts/XTXWPQSEgoMkAupKt/an-intuitive-explanation-of-bayes-s-theorem).
# What trade should we make if we're all getting the new COVID strain?

If data keeps coming out in the next week confirming that [the new COVID strain](https://www.lesswrong.com/posts/CHtwDXy63BsLkQx4n/covid-12-24-we-re-f-ed-it-s-over) is 70% more transmissible, I think the modal outcome is that ~50% of Americans will get it by the early summer. The market may take a few days to realize and react to this (as it was in March), but also just buying June put option on the SP500 seems very naïve (since the SP500 is at all-time highs and a fourth COVID wave doesn't necessarily affect much the NPV of future earning of huge corporations). So if I think that the probability of everyone getting COVID in the next six months is much likelier than the market, at least for a few days, what trade would capture that?
# What are the unwritten rules of academia? 

I came across [this tweet](https://twitter.com/x_lenc/status/1342224583223439360?s=20) a few minutes ago, and it got me thinking about how much of academia is gatekept by unwritten rules and rituals. Are there any good resources out there documenting such rules for outsiders? If not, this seems like a worthy task to attempt for anyone with such knowledge.

Are there any aspects of academia or of doing original research that you would not have known about if you had only learned through textbooks? With my current position of learning exclusively at home without a mentor, I worry that I may be missing many of these unwritten rules, and don't know how much of a disadvantage this places me at.
# What evidence will tell us about the new strain? How are you updating?

I didn't get into this question in the weekly post. The answer does not seem obvious to me.

Eventually, it will be obvious either way. Either the strain will spread rapidly across many locations, or it won't. But we would like to get the information as fast as possible, whatever the answer turns out to be. So the open question is, what numbers or other information can we use in the short term to generate useful evidence for or against the new strain being a lot more infectious than older ones?

The obvious first things to look at are England's case numbers, and how rapidly the new strain continues to outcompete the old strain. Certainly the percent dominance continuing to go up, or stabilizing, is evidence. 

Alas, tonight is Christmas, and I haven't been keeping up in detail with English reporting and testing dynamics. We saw only a small bump on Christmas Eve, after big jumps in the previous post-announcement days, and I can't tell how much of that was the holiday. Same will apply to today's number, to Boxing Day, and then there are issues of either lag or catchup, and then New Year's. Can anyone shed light on these details, and how seriously we should take them? Even if we get a new mix of strain results, should we worry about lags in the south versus the north, where different strains dominate, warping numbers?

What are good data sources to check for the strain in other nations? Who is checking enough that if it was present, we would notice, or we'd have a good handle on its numbers? Can we make this data happen somewhere? 'Has been spotted in' isn't much to work with.

Also making this a place for people to say how they're updating their estimates based on information that comes in.
# Simultaneous Randomized Chess

[David](https://github.com/dchudz) and I played around with the idea of a simultaneous chess variant until we found one that seems balanced. Rules:

1.  Both players secretly select a move.
    1.  If both can execute without conflict, both do.
    2.  If both can execute only in a specific order, they are executed in that order.
    3.  Otherwise, one move is discarded at random and the other executes.
2.  You may not choose a move that would end with your king in check, given the current board position.
3.  You win by capturing or checkmating the opponent's king.
4.  In selecting a move you choose a maximum number of spaces, but a capture before reaching the maximum counts as a successful move.
5.  You may select a move if it could be legal after some opponent move. If it ends up not being legal, it doesn't happen.
6.  To capture en passant you must attempt the capture simultaneously.
7.  If you are using a clock, it only counts the time between the move selections.

Consider this position:

[![](https://www.jefftk.com/king-queen-conflict-resolution.png)](https://www.jefftk.com/king-queen-conflict-resolution-big.png)

Say White chooses f3-f7 and Black anticipates that and chooses e8-f7. Black is allowed to make this move because (2) does not apply and (5) does. If we resolved Black's move first then only White's move could happen, but if we resolve White's first then both can happen, so we get the latter:

[![](https://www.jefftk.com/king-successful-queen-capture.png)](https://www.jefftk.com/king-successful-queen-capture-big.png)

First White takes the pawn at f7, then Black takes the queen at f7. Alternatively, Black could have selected f7-f6. In that case the only way for both to happen would have been for Black's moved to happen first, at which point (4) means White's queen captures at f6 instead of f7:

[![](https://www.jefftk.com/white-queen-captured-pawn.png)](https://www.jefftk.com/white-queen-captured-pawn-big.png)

This is a bad situation for White, because pawns very powerful. Because a diagonal move only succeeds if it is a capture, (1.3) means Black selecting g7-f6 forces a 50-50 for the White queen. Whereas, if the queen were threatened by any other piece it could just move away (1.2).

In writing this up I found that there is already a [Simultaneous Chess](https://www.chessvariants.com/invention/simultaneous-chess) variant. That version has a system of "initiative" to handle conflict resolution, which seems like it would play quite differently.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100198879934852)*
# In software engineering, what are the upper limits of Language-Based Security?

Given the most security-amenable language you know, or that you could imagine building (or given some tools and processes that change the qualities of an existing language), by what proportion could we reduce the incidence rate of major security errors? In what domains could incidence be reduced all the way to zero?

Are there situations where your more secure language can't currently be used, because it imposes higher development or runtime costs?
# Reviews as "How should this be improved to meet the bar?"

The LessWrong Review is a pretty big experiment, and I am still very much uncertain what the best form of reviewing is. We were pretty vague on how to review, listing these bullet points on topics a review should consider:

*   How has this post been useful?
*   How does it connect to the broader intellectual landscape?
*   Is this post epistemically sound?
*   **How could it be improved?**
*   **What further work would you like to see on top of the ideas proposed in this post?**

I'm interested in seeing more of the last two. I'd like a key piece of the Review to be "authors rewriting posts if they can be improved", and "people writing followup posts that either attempt a better explanation of whatever the post was about, or do followup work exploring more of the post's ideas."

So, for negative reviews, if a post's central concept seems important but the post itself feels inadequate, I'd be interested in seeing more "how could this post be improved, such that it makes sense to include in a 'Best Of', or otherwise enter into Lesswrong's longterm memory.*"

* * *

*a concern I have about the current Review process is it focuses primarily on the best-of book, but not all important works make sense to include in a public facing artifact. Examples of this might include a) dry, technical posts, b) posts that are important but somehow controversial, c) posts that are doing a lot of "schlep" work (such as replications), where you don't necessarily care that everyone reads them, but you care that they exist and you can refer to them when doing later meta-analyses or whatnot)

I'm not sure about the best solution for this and this is still the subject of some debate on the LessWrong team, but I'd personally prefer if reviewers somewhat distinguished between "whether it makes sense for a post to be in the Best Of Book" and "whether the post is somehow important to the LessWrong intellectual project."
# Uninformed Elevation of Trust

*I don't know if there is a standard name for this phenomenon (other than the related Gell-Mann amnesia effect, e.g. "The Sequences are great, except in my area of expertise, where they are terrible.").*

Here is the gist: **we trust the data as much as we trust the source, regardless of how much the source trusts the data**.

This sounds unobjectionable on the surface. We tend to equate the reliability of the data with the subjectively perceived trustworthiness of the source of data whenever we have no independent means of checking the veracity of the data. What is lost in this near-automatic logic is one small piece: **the credence the source itself assigns to the data**. 

The (faulty) Bayesian math here is pretty straightforward and is left as an exercise to the reader.

A few examples: 

*   An investment tip from an insider is perceived as reliable even if the insider themselves would consider it a speculation at best.
*   Life on Venus!
*   US election fraud!
*   A tupperware party.
*   A potential hire introduced by a trusted friend or a coworker.
*   ~~A CFAR workshop~~. Just kidding, those are super trustworthy.

I think the titular description, Uninformed Elevation of Trust, tends to capture the essence of what happens, quickly and naturally, whenever we forget or neglect to engage our critical reasoning full on. It's an elevation of trust, not just adjustment of it, because the scenario where we skip the critical evaluation happens more the more we trust the source of data. We just naturally assume that whatever they tell us is as trustworthy as they themselves are, even though it's obvious that it should be discounted by the factor equal to the source's degree of belief in what they tell us. 

Please feel free to offer your own examples and counter examples.
# Infinite possibilities

Naively, for instance from the perspective of me as a child, it seems like a person has vastly many possible options at each moment, leading out in every direction, where many of them surely lead to amazing things, and thus it should be very easy to have an incredibly great life and make a huge positive difference to the world.

The problem with this is that having the ability to do incredible things, and wanting to do those incredible things, is not enough. If you can also do a bazillion other non-incredible things, then you also have to be able to pick out the incredible path from among the rest, and even if you do, a moment later it hits another incomprehensibly complicated intersection of unmarked paths, and you have to do it again.

This perhaps sounds obvious, but I think we do often still talk as if what happens is determined by people’s goals and their capabilities, and ignore the issue of computing which exercise of capabilities will bring about which goals, or leaving it as hopefully irrelevant noise in the model. My tentative guess is that this is a real impediment to thinking about the world and strategizing about life well.

I don’t know if anyone has a better model, or has thought about how bad this is. My tentative guess is that it is bad. It seems like something economists would think about, but I’m not sure what it would be called.
# Where are the post-COVID complainers?

Assumption: COVID immunity lasts a long time, because that's how diseases work, and this is the case for 95+% of people who get COVID or a COVID vaccine.

20% of the United States has had COVID (13-30% per https://covid19-projections.com/) and presumed immune. Even if we conservatively take the official case count (20M) and trim off 25% to account for possible duplicates, that's 5% of the population.

Another 1-2% has already been vaccinated and will join them in a month, with more every week after that.

So on the low end, in mid-January, 7% of the country will be fine. COVID-fine, at any rate, but still subject to all local, statewide, and national restrictions.

Why aren't these people making noise about going back to their normal lives? They want to eat in restaurants and make money working in restaurants. They want to see their friends and shop. They want to do lots of things.

I notice that I am very surprised that this large and growing minority isn't insisting on partial re-openings (with immunity certificates?). I'm surprised they're not screaming for it. I'm surprised it's not a discussion all across the media. What am I missing?
# Measuring up to incredible potential

Yesterday [I wrote](https://worldspiritsockpuppet.com/2020/12/28/infinite-possibilities.html) that people often talk as if events are basically determined by people’s values and capabilities, ignoring the difficulty of figuring out which opportunities to take, or even noticing opportunities.

I think one reason to have a better model is that this one doesn’t account for a substantial category of felt difficulty in being a human, possibly encouraging a general sense that one is ubiquitously failing, what with not seeming to be demonstrably grabbing the best of a vast multitude of possible options at each moment.

My own experience for instance involves often not remembering even the options that I noticed before, and these not seeming that multitudinous, except relative to my pitiful capacity to hold things in my mind. Also, regularly noticing new aspects of the world suggestive of many options whose existence hadn’t even dawned on me previously, even though they would be pretty obvious to a creature for whom logic was free. And mostly being extremely uncertain about what actions will lead to what outcomes. If I habitually model myself as roughly _homo economicus_, deftly directing my resources toward my goals, at worst according to my pristine , it seems like I am in danger of continually find my real self incredibly disappointing. Plus, abstracting away parts of the situation that might be rich with potential for improvement.
# Why Neural Networks Generalise, and Why They Are (Kind of) Bayesian

Currently, we do not have a good theoretical understanding of how or why neural networks actually work. For example, we know that large neural networks are sufficiently expressive to compute almost any kind of function. Moreover, most functions that fit a given set of training data will not generalise well to new data. And yet, if we train a neural network we will usually obtain a function that gives good generalisation. What is the mechanism behind this phenomenon?

There has been some recent research which (I believe) sheds some light on this issue. I would like to call attention to this blog post:

[Neural Networks Are Fundamentally Bayesian](https://towardsdatascience.com/neural-networks-are-fundamentally-bayesian-bee9a172fad8 )

This post provides a summary of the research in these three papers, which provide a candidate for a theory of generalisation:

[https://arxiv.org/abs/2006.15191](https://arxiv.org/abs/2006.15191)   
[https://arxiv.org/abs/1909.11522](https://arxiv.org/abs/1909.11522)   
[https://arxiv.org/abs/1805.08522](https://arxiv.org/abs/1805.08522)  
  
(You may notice that I had some involvement with this research, but the main credit should go to Chris Mingard and Guillermo Valle-Perez!)  
  
I believe that research of this type is very relevant for AI alignment. It seems quite plausible that neural networks, or something similar to them, will be used as a component of AGI. If that is the case, then we want to be able to reliably predict and reason about how neural networks behave in new situations, and how they interact with other systems, and it is hard to imagine how that would be possible without a deep understanding of the dynamics at play when neural networks learn from data. Understanding their inductive bias seems particularly important, since this is the key to understanding everything from [why they work in the first place](https://arxiv.org/abs/1611.03530), to phenomena such as [adversarial examples](https://openai.com/blog/adversarial-example-research/), to the risk of [mesa-optimisation](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB/p/q2rCMHNXazALgQpGH). I hence believe that it makes sense for alignment researchers to keep an eye on what is happening in this space.  
  
If you want some more stuff to read in this genre, I can also recommend these two posts:  
  
[Recent Progress in the Theory of Neural Networks](https://www.lesswrong.com/posts/KrQvZM8uFjSTJ7hq3)  
[Understanding "Deep Double Descent"](https://www.alignmentforum.org/posts/FRv7ryoqtvSuqBxuT/understanding-deep-double-descent )  
  
EDIT: Here is a second post, which talks more about the "prior" of neural networks:  
  
[Deep Neural Networks are biased, at initialisation, towards simple functions](https://towardsdatascience.com/deep-neural-networks-are-biased-at-initialisation-towards-simple-functions-a63487edcb99)
# Dario Amodei leaves OpenAI

This is a linkpost for [https://openai.com/blog/organizational-update/](https://openai.com/blog/organizational-update/)

> “We are incredibly thankful to Dario for his contributions over the past four and a half years. We wish him and his co-founders all the best in their new project, and we look forward to a collaborative relationship with them for years to come,” said OpenAI chief executive Sam Altman.

Anyone know what the new project is?
# What Are Some Alternative Approaches to Understanding Agency/Intelligence?

Apparently, MIRI has [given up](https://intelligence.org/2020/12/21/2020-updates-and-strategy/) on their current mainline approach to understanding agency and are trying to figure out what to do next. It seems like it might be worthwhile to collect some alternative approaches to the problem -- after all, intelligence and agency feature in pretty much all areas of human thought and action, so the space of possible ways to make progress should be pretty vast. By no means is it exhausted by the mathematical analysis of thought experiments! What are people's best ideas?

(By 'understanding agency' I mean research that is attempting to establish a better understanding of how agency works, not alignment research in general. So IDA would not be considered agent foundations, since it takes ML capabilities as a black-box. )

ETA: I originally wrote 'agent foundations' in place of 'understanding agency' in the above, which was ambiguous between a broad sense of the term(any research aimed at obtaining a foundational understanding of agency) and a narrow sense(the set of research directions outlined in the agent foundations agenda document). See [this comment](https://www.lesswrong.com/posts/ppmtsKiodNceMe3k6/what-are-some-alternative-approaches-to-agent-foundations?commentId=ZMpH4vftP6A8ASDnb) by Rob re: MIRI's ongoing work on agent foundations(narrow sense).
# Desires as film posters

Sometimes I like to think of desires as like film posters. You come across them, and they urge you to do something, and present it in a certain way, and induce some inclination to do it. But film posters are totally different from films. If you like a film poster, you don’t have to try to see the film. There is no metaphysical connection between the beauty of a film poster and the correctness of you seeing the film. It’s some evidence, but you have other evidence, and you get to choose. A film poster can be genuinely the most beautiful film poster you’ve ever seen, without the film being a worthwhile use of two hours. That’s largely an orthogonal question. If you put up the poster on your wall and look at it lovingly every day, and never see the film, that doesn’t need to be disappointing—it might be the best choice, and you might be satisfied in choosing it.
# Review Voting Thread

We're half way through the second annual review, and 121 posts have been nominated

We've had more than double the number of individual nominations than last year, but on reviews we're still playing catchup. Last year we had 118 reviews, yet this year we've only had 51 so far.

When there's so many posts, it can be daunting to figure out which ones to review, so to help out, I'm making this thread. **Every comment on this thread will be a post, and you should vote on which ones you would like to read a review of. **

A review is something ideally that puts it in context of a broader conversation, describes its key contributions, its strengths and flaws, and where more work can be done.

(Or something else. Many people who write self-reviews often give a different flavor of review. And I've read many great short reviews, e.g. Jameson Quinn and Zvi last year did a lot of short reviews that communicated their impression of the post quite clearly.)

So I'm going to leave 122 comments on this post. 121 comments will just be a post title, and the other one will be for thread meta. (Search "Meta Thread".) I will remove my own votes from them, so they all start at zero. 

Please vote on the comments to show how much you'd like to see reviews of different posts! Feel free to add a comment about what sort of review you'd like to see.

(Yes, I will probably get a lot of karma from this thread. Mwahaha you have fallen for my evil trap.)

(Also, my thanks to reviewers magfrump and Zvi with 5 each, johnswentworth with 6 reviews, and to fiddler with 10 (!), all thoughtful and valuable.)
# And You Take Me the Way I Am

Mark Twain [wrote that](https://www.goodreads.com/quotes/9131-if-you-tell-the-truth-you-don-t-have-to-remember) honesty means you don't have to remember anything. But it also means you don't have to worry about making mistakes.

If you said something terrible that made everyone decide that you're stupid and evil, there's no sense in futilely protesting that "that's not what you [meant](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie)", or agonizing that you should have thought more carefully and said something else in order to avoid the outcome of everyone thinking that you're stupid and evil.

_Strategy is deception_. You said what you said in the situation you were in, and everyone else used [the information in that signal](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist) as evidence for a Bayesian update about your intelligence and moral character. As they should. So what's the problem? You wouldn't want people to have _false beliefs_, would you!?
# Review: The Diving Bell and the Butterfly

I suspect my mind of taking its observations of a person’s physical energy and dexterity as strong evidence about their mental quickness and clarity.

The existence and the wrongness of this presumption were brought into relief for me by reading Jean-Dominique Bauby’s _The Diving Bell and the Butterfly_, on his life with locked-in syndrome. Because realizing that the author’s lively and intelligent voice was issued from a single blinking eye looking out of a mostly inert body felt like seeing a magic trick.

But perhaps it is just that a writing process without vigorous back-and-forth with a mind-expanding piece of paper or virtual paper—a place to lay out one’s words and see them and move them around, without having to keep the entire gameboard in one’s head—sounds mentally paralyzing to me.

At this point, I realize that in all likelihood it is my own mind that is weak and wasted from clinging to these crutches, and that has learned to fear starting out by itself with no piece of paper to reassuringly catch its thoughts and promise that nothing will be forgotten. Bauby’s mind may well have become stronger between his time as eloquent editor of French _Elle_ and writing this book.

It probably isn’t the feat of its creation though that makes this enjoyable to read, or makes my mind keep coming back to it. It’s a bright and compelling window into another person’s mind. (The bizarre tragedy of the plot probably doesn’t hurt either.)

Another thing I found interesting about the book was that the setting of being immobile in a hospital bed doesn’t give the reader many of the clues they might usually use to make out the character of the protagonist. How does he hold himself, when he can hold himself? How does he talk to his partner, when he can do more than blink? What did he want in life, before the goalposts were changed and, as he says, not drooling all the time became a cherished ambition? (We do learn that he is 44 years old, and that if he is going to drool, he would prefer to drool on cashmere.) Yet, I did not have a sense of blankness about his personality. From his narration of the world, I felt that I knew him. However toward the end, he narrates the final day before the stroke that detached his mind and his body mostly. And I realize that if I had met him in that way first, I would have a different impression of him. And I liked the way around that I did meet him. So I wonder, if it would be better to more often meet people’s trains of thought for a while before getting to see their appearance or overt behaviors.

I composed each paragraph of this post in my head except about fifteen words of editing, and dictated the first line to my boyfriend via winking.
# Data about the new coronavirus variant (B.1.1.7) from Denmark

I noted on Zvi's recent post (found here [https://www.lesswrong.com/posts/gbuXXC9KvphmXQFLd/what-evidence-will-tell-us-about-the-new-strain-how-are-you#GxzkzNdYfTLjmQEAm](https://www.lesswrong.com/posts/gbuXXC9KvphmXQFLd/what-evidence-will-tell-us-about-the-new-strain-how-are-you#GxzkzNdYfTLjmQEAm) ) that Denmark was one of few other nations to do large amounts of Covid  strain sequencing.  
This is a followup with the newly released data.  
The data seems consistent with a doubling time around a week.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2fd259aca01c856e2f314620cbeb28504a886b7dff277c27.png)

Data found here [https://www.ssi.dk/-/media/cdn/files/opdaterede-data-paa-ny-engelsk-virusvariant-sarscov2-cluster-b117--01012021.pdf?la=da](https://www.ssi.dk/-/media/cdn/files/opdaterede-data-paa-ny-engelsk-virusvariant-sarscov2-cluster-b117--01012021.pdf?la=da)   
 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b9608f9cb43e15857c132c8d567683add7bbfd05c859a063.png)

Here is a graph over cases which where tested on specific days, note that days where there where 0 tests sequenced as B.1.1.7 have been skipped  
(colors corespond to different areas in Denmark)
# Feature request: personal notes about other users

Feature use scenario:

*   I hover the mouse pointer over a username
*   The small box that appears (the one that currently shows basic stats: when they joined, how many posts they have written, etc.) contains also a textarea field
*   Whatever I write in that field persists between my logins, until I edit it again (if ever)
*   The contents of that field are only visible to me

Motivation: As I read LW more, I'm starting to attribute expertise in some subjects to some users. And conversely, I'm noticing specific biases in other users. I would like to privately keep note of that, so that I can more precisely trust/distrust what they write in the future.
# Bucket Brigade: Video

The biggest pain point with Bucket Brigade has been that you generally wanted to be running a video call in parallel. You kept needing to switch windows, and you needed to remember to mute and unmute in the call at the beginning and end of each song. [Solstice](https://www.jefftk.com/p/secular-solstice-2020) used [Twilio Video](https://www.twilio.com/video), which packages up WebRTC with server-side support and a developer-friendly API, and I decided to integrate it as well. Now when you join you'll see it ask for camera permissions:

[![](https://www.jefftk.com/bucket-brigade-video-permission.png)](https://www.jefftk.com/bucket-brigade-video-permission-big.png)

Before showing your video to anyone else, it will ask you whether you are OK with it:

[![](https://www.jefftk.com/bucket-brigade-choose-camera.png)](https://www.jefftk.com/bucket-brigade-choose-camera-big.png)

Since I have an external camera with a better angle, I'll press "Switch camera":

[![](https://www.jefftk.com/bucket-brigade-other-camera.png)](https://www.jefftk.com/bucket-brigade-other-camera-big.png)

Now I'm happy, so I'll choose "Use this camera", but it will also work fine if you choose "Continue without camera".

Once you're in, you'll see all the people in their buckets:

[![](https://www.jefftk.com/bucket-brigade-video-clones.png)](https://www.jefftk.com/bucket-brigade-video-clones-big.png)

During a song you hear people via Bucket Brigade, and the rest of the time you'll hear people through the video call. Since we aren't doing anything custom for the videos, you'll see people out of sync with their singing. At some point it would be nice to try routing videos through a server we control, so we can add appropriate amounts of latency. Adding latency on the server would not be precise enough for audio, which we need aligned at the millisecond level, but I think it would be ok for video? Of course there's nothing you can do for video for buckets after you...

The hardest part of the integration was getting it to behave correctly around people turning their video on and off. Twilio's recommended approach doesn't turn the camera light off when you turn your video off, which seems broken to me, so I took their [not recommended approach](https://www.twilio.com/docs/video/javascript-getting-started#mute-your-local-media) of unpublishing video tracks. This changed which events I need to listen to for video being re-enabled, but once I got it all working it seems reliable.

Previously Bucket Brigade was cheap to run: it essentially only needs CPU and bandwidth, and the bottom-tier VPS (~$5/month) that powers jefftk.com can handle about 40 simultaneous users. Twilio calls cost [25¢/person/hour](https://www.twilio.com/video/pricing), which is reasonable but not negligible. I'm hoping to keep it free to play with, and if the cost starts adding up we'll probably ask if frequent users could chip in?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100199706977452)*