Harley-ml/lesswrong huggingface dataset:
# Announcing the Vitalik Buterin Fellowships in AI Existential Safety!

_Epistemic status: describing fellowships that I am helping with the administration of._

_Edit 2021-10-04: Modified to reflect changed eligibility+stipend conditions._

The Future of Life Institute is launching new [PhD](https://grants.futureoflife.org/prog/phd_fellowship/) and [postdoctoral](https://grants.futureoflife.org/prog/postdoctoral/) fellowships to study [AI existential safety](https://grants.futureoflife.org/res/p/AI/): that is, research that analyzes the most probable ways in which AI technology could cause an existential catastrophe, and which types of research could minimize existential risk; and technical research which could, if successful, assist humanity in reducing the existential risk posed by highly impactful AI technology to extremely low levels.

The [Vitalik Buterin PhD Fellowship in AI Existential Safety](https://grants.futureoflife.org/res/p/phd-fellowship-ai/) is targeted at students applying to start their PhD in 2022, or existing PhD students who would not otherwise have funding to work on AI existential safety research. Quoting from the page:
> At universities in the US, UK, or Canada, annual funding will cover tuition, fees, and the stipend of the student's PhD program up to $40,000, as well as a fund of $10,000 that can be used for research-related expenses such as travel and computing. At universities not in the US, UK or Canada, the stipend amount will be adjusted to match local conditions. Fellows will also be invited to workshops where they will be able to interact with other researchers in the field.
>
> In addition, applicants who are short-listed for the Fellowship will be reimbursed for application fees for up to 5 PhD programs, and will be invited to an information session about research groups that can serve as good homes for AI existential safety research.

Applications for the PhD fellowship close on October the 29th.

The [Vitalik Buterin Postdoctoral Fellowship in AI Existential Safety](https://grants.futureoflife.org/res/p/postdoc-fellowship-ai/) is for postdoctoral appointments starting in fall 2022. Quoting from the page:
> For host institutions in the US, UK, or Canada, the Fellowship includes an annual $80,000 stipend and a fund of up to $10,000 that can be used for research-related expenses such as travel and computing. At universities not in the US, UK or Canada, the fellowship amount will be adjusted to match local conditions.

Applications for the postdoctoral fellowship close on November the 5th.

You can apply at [grants.futureoflife.org](https://grants.futureoflife.org/), and if you know people who may be good fits, please help spread the word!
# How do you learn to take more beautiful pictures with a camera?

When I see professionally made photos, I can see that they look different and better then pictures I randomly take. I however have no idea how to aquire that skill. Has anyone here learned how to take good photographs and an idea of what's the easiest way to learn it?
# How dangerous is Long COVID for kids?

All adults in my family are already vaccinated. But I also have two kids, age 6 and 3, who are not. The older one needs to wait a few months until the child vaccine is approved. The younger one will probably have to wait more than a year, because as far as I know the child COVID vaccines are only planned for kids 5 years or older.

How much should I be afraid until my kids get vaccinated? How much precaution is reasonable?

Most online debates about "kids and COVID" focus on the fact that the kids have a very small chance to *die* from COVID. There are numbers and pictures showing that an unvaccinated X years old child actually has a *smaller* chance of dying than a vaccinated Y years old adult. Okay, I got the memo.

But the number of people who end up in hospital and *don't* die is much larger than the number of people who die, right? My friends had their child for one month in a hospital, for COVID-related reasons, and although the child seems perfectly healthy now, I'd prefer to avoid *this* kind of experience, too. How high is the probability of "kid ends up in hospital (but survives)"? I don't have good data here: some people keep sending me articles about crowded child hospitals, other people insist that those were children with preexisting medical problems so this doesn't apply to my kids who seem average health-wise.

Plus there is the question of Long COVID. Kids on average have *more* years of life ahead of them, so even if the chance of a problem is smaller, if it is a permanent damage, you need to multiply it by a greater number to get the *total* harm. Here the data is even noisier, I guess. First, given that we only know COVID for two years (and it keeps mutating), it is hard to estimate whether the damage is actually permanent, or how long it takes. Second, the symptoms of Long COVID are various, and they happen with other diseases too, so I guess in all studies you also have people with symptoms of "Long COVID" who actually never had COVID. So, what is the actual risk distribution here, for kids?

I assume that when kids sit all day long in a classroom, it doesn't make much difference whether they wear masks on not; ultimately they are breathing the same air for hours in a badly ventilated room. Plus they remove the masks in the canteen, or when eating snack. What are the expected consequences of exposing my children to this situation, as opposed to keeping them at home until the vaccine is ready?
# Walkie-Talkies

In the discussion on [kids roaming](https://www.jefftk.com/p/kids-roaming) a few weeks ago, Cullyn [suggested](https://www.jefftk.com/p/kids-roaming#fb-10100228286039782) walkie-talkies as a way to keep in touch with kids who are out of sight. This wasn't something I had thought about at all: my only experience had been with short-range children's models in the early '90s, but now you can get rechargeable ones that will reach about 1/4mi through houses for [~$13/each](https://smile.amazon.com/gp/product/B08MKT9B7X/) shipped.

[![](https://www.jefftk.com/walkie-talkie.jpg)](https://www.jefftk.com/walkie-talkie-big.jpg)

I got a set of four a little over a week ago, and we've been trying them out:

*   We started with a lot of playing with them in situations where they're not really needed, with the kids running ahead or something.
    
*   Before using them each time, we test them to make sure we have them set up right.
    
*   Other kids want to play with the walkie-talkies, but will be rough and change the settings. We ended up needing to make a rule that they weren't for sharing.
    
*   We've practiced setting the channel and volume several times, in case they do get messed up.
    
*   While I can clip mine onto my belt, or put it in my pocket, the kids rarely have appropriate clothes for that. Julia got a small fanny pack for Lily, and tweaked it to fit better.
    
*   Initially, we're only using them for short times and distances, at a range where I would be OK just walking over and checking on them every few minutes.
    
*   Lily (7y) is much better at using them than Anna (5y). I don't think I would let Anna go very far with one unless it was to meet Lily.
    
*   These particular ones speak their settings out loud as you change them, which the kids really like. On the other hand, the knobs are quite hard to read, so Julia painted a white stripe on each so you can see their setting at a glance.
    

We tested the range, and in our flat neighborhood of densely packed three-story wooden buildings it was clear at 1,200ft, down to almost unusably fuzzy at 1,500ft. This is farther than we are OK with the kids going, and farther than their friends' houses, so I don't expect it to be an issue for a while.

Before deciding to get these, we had also considered a watch with cellphone + GPS like the [Gizmo](https://www.verizon.com/connected-smartwatches/verizon-gizmowatch-2/). That seems like a nice idea, and I liked the idea of GPS, but it's more fragile, more expensive up front, and has ongoing costs. Since the walkie-talkie was so much cheaper, we decided to give that a try and see how we liked it. So far so good!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100230128896682)*
# Cartesian Frames and Factored Sets on ArXiv

Papers on Cartesian frames and factored sets are now on arXiv. 

Cartesian Frames: [https://arxiv.org/abs/2109.10996](https://arxiv.org/abs/2109.10996)

Factored Sets: [https://arxiv.org/abs/2109.11513](https://arxiv.org/abs/2109.11513)

The factored set paper is approximately identical to the sequence [here](https://www.lesswrong.com/s/kxs3eeEti9ouwWFzr), while the Cartesian frame paper is rewritten by Daniel Hermann and Josiah Lopez-Wild, optimized for an audience of philosophers.
# This War of Mine

*This War of Mine* is too hard.

There are many kinds of war videogames on the market. First person shooters let you play as a marine, a robot soldier, a WWI soldier, a WWII soldier, a WWIII soldier or an alien soldier. Strategy games let you command armies, squads, nations and empires. You can kill Nazis, robots, ninjas, Rebels and Communists. You can shoot guns, wield swords, fire rocket launchers, operate flamethrowers and throw grenades. You can fly aircraft, command tanks, drive trucks, race humvees, pilot battlemechs and sail ships.

*This War of Mine* is based on the Siege of Sarajevo. You play as a noncombatant trying to not die.

You can burn your books to keep from freezing to death but if you do your avatar might get bored from having nothing to read. You can trade knives for food and diamonds for cigarettes. If you make a chair it'll save you energy during the day. If you make a bed you will be more likely to survive getting sick. But if you build a bed that's supplies you can't use for boarding up your windows or constructing a system to collect rainwater.

The constant risk of starving to permadeath is not what makes *This War of Mine* difficult.

Early in *This War of Mine* you come across a house while scavenging. There is an elderly couple living there. Do you steal the old man's medicine? You live a precarious existence. Small mistakes compound. Your friends are counting on you to bring back food and other desperately-needed supplies. Being a little hungry one day could cause a cascade of problems resulting in all of you dead.

On the other hand, stealing an old man's medicine is wrong.

The first time I played *This War of Mine* I thought long and hard about what to do. I turned the game off and went for a walk. The tenth time I stabbed the old man with a knife while his back was turned so I could practice the game's stealth mechanics.

There are [opportunities to do good](https://www.lesswrong.com/posts/3CxbpKnpN4BHrwoXn/hell-is-wasted-on-the-evil) too. A young woman might come by and ask for help boarding up her house to protect against scavengers. If you accept it'll cost you a day's labor. You'll be a day hungrier with nothing to show for it.

If you accept the moral boost might keep your character from committing suicide.

In *This War of Mine* you're repeatedly forced to choose between risking your life and risking your soul. You don't know how long the siege will last. It could end tomorrow or it could end in a year. It's a challenge to survive two weeks in *This War of Mine*. The real world siege lasted nearly four years.

If, somehow, you survive to the end there's no score. Instead, the game shows you photos of  the strangers you helped along the way.
# Secular Therapy Project

>  The purpose of the Secular Therapy Project is to help connect non-religious or secular persons who need mental health services with outstanding mental health professionals, such as psychologists, psychiatrists, counselors, and others. Using our system is simple and requires very little information from you. Our goal is to protect your confidentiality until you find a therapist to correspond with or to work with.
>
> What’s unique about the STP is that we aren’t just a database of therapists. Instead, we very carefully screen potential therapists who want to become part of the STP. We screen them to make sure that a) they are appropriately licensed in their state or country, b) that they are secular in nature as well as practice, and c) that they actually use evidence-based treatments, which have been shown to be effective at helping improve mental health problems in controlled clinical trials. This means not only will our therapists not try to preach to you or convert you, but that they are also using the most well-supported types of treatment to help you.
# Schools probably do do something

It's a relatively commonly held view here on Less Wrong that school is mostly a waste of time.

I'm not going to comment on this directly. Instead I'm going to show that school probably has some impact on future life outcomes.

The evidence for this is the [Relative Age Effect](https://en.m.wikipedia.org/wiki/Relative_age_effect).

> The term relative age effect (RAE), also known as birthdate effect or birth date effect, is used to describe a bias, evident in the upper echelons of youth sport and academia, where participation is higher amongst those born early in the relevant selection period (and correspondingly lower amongst those born late in the selection period) than would be expected from the normalised distribution of live births. The selection period is usually the calendar year, the academic year or the sporting season.

For example students in Oxford were well over 20% more likely to be born in September (the start of the academic year), than August (the end).

The classic explanation for this is that whilst there's not much of a difference in maturity between a 17 year old and an 18 year old, there's a large difference between a 4 year old and a 5 year old. So older kids shoot to the top of the class when they start school, and so are more likely to be engaged and interested in class, which keeps them at the top even once the difference in maturity stops being relevant.

This implies that either

A) School does something positive for those at the top of the class (pushes them up relative to kids who don't go to school at all).

B) School demotivates those at the bottom of the class (pushes them down relative to kids who don't go to school at all).

C) Given how difficult it is to get into Oxford, even 1 years worth of maturity can make a significant difference.

If we discount C, then school does have some impact on academic outcomes albeit we don't know if that outcome is positive or negative.

Further research could include:

- is there a difference between students who took a gap year before applying and those who didn't? This would investigate option C above.
- does this effect persist among home schooled children, where the classic explanation wouldn't apply.
- does this effect persist into later life outcomes? Thai would measure whether the impact is short term or long term. According to [Wikipedia](https://en.m.wikipedia.org/wiki/Relative_age_effect#In_leadership_positions) this effect has been observed among CEOs of S&P 500 companies as well. However this is maybe just because they're more likely to get into a prestigious university, which is what really matters, not that school directly had an impact.
# Chris Dixon's Crypto Claims are Logically Flimsy

In this post, I make heavy use of [the power to demolish bad arguments](https://www.lesswrong.com/posts/FfrWGCEhZJbkeFgww/the-power-to-demolish-bad-arguments) in the domain of crypto claims. The crypto world seems rich with areas where you can save a lot of wasted time and money by simply asking for a specific example of how a crypto project beats the best available alternative from the non-crypto world.
# The Trolley Problem

If you go to Harvard Law School, one of the first classes you'll take is called "Justice: What's The Right Thing To Do?" It starts with the trolley problem.

> There's a trolley hurling down the tracks with no breaks. It's on its way to hitting and killing 5 people. You can pull a lever but if you do the trolley will kill 1 person instead. Do you pull the lever?

Students are asked "Should you pull the lever?"
- Some say "Yes because utilitarianism should maximize human life."
- Some say "No because the categorical imperative forbids murder."

They're both wrong. The question is underspecified. The right answer is "It depends."

> You're in Germany in 1943. A train full of political prisoners is hurtling toward its doom. You can pull a lever to save their lives but if you do the train will hit Heinrich Himmler instead. Do you pull the lever?

Heck yeah!

> You're a fighter in the French Resistance in 1943. A train full of SS officers is hurtling toward its doom. You can pull a lever to save their lives but if you do it'll hit your friend who is planting a bomb on the railroad tracks. Do you pull the lever?

Of course not.

Those hypothetical situations cheat the problem. The purpose of the trolley problem is to compare one positive value against another positive value. Let's do that.

> You're a general in the Russian army in 1943. A train full of Russian soldiers is hurtling toward its doom. You can pull a lever to save their lives but if you do it'll hit a Russian soldier standing on the tracks. All the soldiers are of the same rank. Do you pull the lever?

Yes, obviously. Your job is to maximize the combat effectiveness of your fighting force.

> You're a doctor in 2021. Do you murder a patient and harvest his organs to save the lives of 5 other people?

No, obviously. That would be monstrous. Even if it weren't, the negative externalities would overwhelm your proximate benefit.

If the answer to an ethical question is always the same, regardless of context, then you can reason about it as a context-free abstraction. But answers to ethical questions are usually context-dependent.

If I took the class I'd ask "What year is it? Where am I? Who am I? Who's in the trolley? Who's on the tracks? Who designed the trolley? Who is responsible for the brake failure? Do I work for the trolley company? If so, what are its standard operating procedures for this situation? What would my family think? Would either decision affect my future job prospects? Is there a way for me to fix the systemic problem of trolleys crashing in thought experiments? Can I film the crash and post the video online?"

[![](https://s3.us-west-2.amazonaws.com/www.lsusr.com/lesswrong/trolley-problem.png)](https://xkcd.com/1455/)

By the time I get answers to half those questions, the trolley will have crashed, thus resolving the dilemma.
# Is progress in ML-assisted theorem-proving beneficial?

I want to believe that the outputs of the recently founded [Hoskinson Center for Formal Mathematics](https://www.cmu.edu/news/stories/archives/2021/september/hoskinson-center-for-formal-mathematics.html) (led by [Jeremy Avigad](https://www.cmu.edu/dietrich/philosophy/people/faculty/jeremy-avigad.html), researcher of mathematical logic, automated reasoning and philosophy of mathematics), will live up to Mr Hoskinson's aura of technical responsibility.

One of the stated goals of the HCFM is to make mathematics more legible (both between mathematicians, and to non-mathematicians), and since mathematics is itself a motor of legibility in software systems and physical engineering, maybe new techniques in automated proving will make problems in AI motivation more legible too. That would be really great.

On the other hand, I don't know how to place bounds on the set of problems that could be solved with superhuman mathematical reasoning. HCFM is likely to produce a broad set of unexpected capabilities, and I can't see a way to exclude the problem of real-world, agentic utility optimization from that set.

One of the limitations of neural models right now is that they have only very limited kind of memory, they're mostly just reflexes, very wise ones, evolved over the course of, often the equivalent of tens of thousands of years of condensed human experience, but none of these wise reflexes so far can evolve a set of coherent thoughts over time, pretty much everything they do is an instant reaction to the current state of their environment.  
  
GPT3 was startling because words are a type of memory, and our writing often depicts evolving thoughts, so sometimes it verges on evolving thoughts, but it was a limited kind of thinking because words are imprecise and most of the training data was rambly and inconsequential. That said, I don't know what GPT3's limits would turn out to be after, for instance, being subjected fully to [Anthropic](https://www.anthropic.com/)'s inspection technique, and the tuning technique that will grow from that.  
  
Well, mathematical claims constitute an extremely precise and robust format for the encoding of memories. In the training data, formal proofs, reference correlates closely with salience; the components are cited, new and interesting fruits always follow from them. To evolve the ability to predict the conclusions that follow the assumptions is equivalent to evolving the mind of a mathematician, in their prime.  
  
Each mathematical result is a kind of evolution of a kind of thought, older thoughts evolving creatively and usefully into new ones. For me, there is no ambiguity as to whether artificial mathematicians would be thinking machines, and it's difficult for me to discern the limits of what this style of thought, sped up and scaled horizontally, would be able to think about.
# Collection of arguments to expect (outer and inner) alignment failure?

Various arguments have been made for why advanced AI systems will plausibly not have the goals their operators intended them to have (due to either [outer](https://www.lesswrong.com/tag/outer-alignment) or [inner](https://www.lesswrong.com/tag/inner-alignment) alignment failure).

I would really like a **distilled collection of the strongest arguments**.

Does anyone know if this has been done?

If not, I might try to make it. So, any replies pointing me to resources with arguments that I've missed (in [my own answer](https://www.lesswrong.com/posts/m5or8yzrw9GLavz9b/collection-of-arguments-to-expect-outer-and-inner-alignment?commentId=E7pf5p69LD2PAoMrx)) would also be much appreciated!

Clarification: I'm most interested in arguments that alignment failure is *plausible*, rather than merely that it is *possible* (there are already examples that establish the *possibility* of [outer](https://openai.com/blog/faulty-reward-functions/) and [inner](https://www.lesswrong.com/posts/iJDmL7HJtN5CYKReM/empirical-observations-of-objective-robustness-failures) alignment failure for current ML systems, which probably implies we can't rule it out for more advanced versions of these systems either).
# You can talk to EA Funds before applying

_Cross-posted to the [EA Forum](https://forum.effectivealtruism.org/posts/htrccjrwLLfC7NAn9/you-can-talk-to-ea-funds-before-applying)._

One thing I have realized as a fund manager for the [EA Long-Term Future Fund](https://funds.effectivealtruism.org/funds/far-future) is that there are a lot of grants that I would like to make that never cross our desk, just because potential applicants are too intimidated by us or don’t realize that their idea is one we’d be willing to fund if only they applied.

To try to remedy that problem, I’m going to start offering the following service: if you have any idea of any way in which you think you could use money to help the long-term future, but aren’t currently planning on applying for a grant from any grant-making organization, I want to hear about it. Feel free to send me a private message on the [EA Forum](https://forum.effectivealtruism.org/users/evhub) or [LessWrong](https://www.lesswrong.com/users/evhub). I promise I’m not that intimidating :)

Not only that, but having talked about this with some of the other [EA Funds](https://funds.effectivealtruism.org/) managers, many of them were willing to extend the same offer as well:

* For the [Animal Welfare Fund](https://funds.effectivealtruism.org/funds/animal-welfare), just [send an email to Kieran Greig](mailto:kieran@effectivealtruismfunds.org).
* For the [EA Infrastructure Fund](https://funds.effectivealtruism.org/funds/ea-community), both Michelle Hutchinson (via [EA Forum](https://forum.effectivealtruism.org/users/michelle_hutchinson) or [email](mailto:michelle@effectivealtruism.org)) and Michael Aird (currently a guest manager; can be contacted via [EA Forum](https://forum.effectivealtruism.org/users/michaela), [LessWrong](https://www.lesswrong.com/users/michaela), or [email](mailto:michaeljamesaird@gmail.com)) would be happy to chat.

For example, here are some of the sorts of grants I’m often excited about but that I rarely see anyone apply for:

* “I want to transition to a career in something longtermist, but that transition would be difficult for me financially and I’d like to have some extra financial reserves to make it easier.”
* “I think I would be more productive in my longtermist job if I had more money to spend on things that would save me time.”
* “I have an idea for a longtermist project I want to work on, but I don’t want to commit to definitely working on that project for the duration of a long grant and want freedom to change my mind and switch to a different project if I want.”
* “I have an ambitious idea for a project that I think would benefit the long-term future, but I think it would take a lot of money, more than what I normally see LTFF grants being given out for.”

Really, though, I don’t want to anchor anybody too much on these specific ideas—if you have any idea of any way in which you think you could use money to help the long-term future, I want to hear about it.
# [Prediction] We are in an Algorithmic Overhang

The primary purpose of this blog post is to create a public record. Some technical insights are deliberately omitted.

I think we're in an algorithmic AI overhang.
1. Today's neural networks (including GPT) are so data inefficient they will not strictly outcompete human brains' performance in all domains (excluding robotics, which is hardware-limited) no matter how much data we shove into them and how big we scale them.
2. The human brain uses a radically different core learning algorithm that scales much better as a function of its training data size.
3. The core learning algorithm of human beings could be written in a handful of scientific papers comparable to the length and complexity of Einstein's *Annus Mirabilis*.
4. Once the mathematics behind the human brain's learning algorithms are made public, they will be running at scale on silicon computers in less than 10 years.
5. Within 10 years of getting these algorithms to scale, they will be cheap enough for a venture-backed startup to run them at a scale outstripping the smartest human alive―assuming civilization lasts that long. (A world war could destroy the world's semiconductor fabricators.)

I've been thinking about this idea for a long time. What finally pushed me to publish were two fire alarms in sequence. First, a well-respected industry in the leader of AI stated in a private conversation that he believed we were algorithmically limited. Secondly, Steven Byrnes wrote [this post](https://www.lesswrong.com/posts/W6wBmQheDiFmfJqZy/brain-inspired-agi-and-the-lifetime-anchor). The basilisk is out of Pandora's Box.

---

Part 2 [here](https://www.lesswrong.com/posts/dY9k9d6cLzqx7FJ6P/prediction-we-are-in-an-algorithmic-overhang-part-2)
# How can one train philosophical skill?

I'm not sure whether this is better-framed as a babble question or a "serious" question, so, treat it as you will. 

If you could sit down and train in philosophy the way you train in arts or sports or maths, how would you do it? What's the Deliberate Practice of philosophy look like? What would a philosophy coach do? Would there be philosophy competitions? What could they look like?

Or, if you think philosophy as it is currently practiced is useless/suboptimal, what might you put in its place?
# Porchfest 2021

Yesterday, I played at Somerville Porchfest with several contradance friends. This is an annual festival that has been running since 2011, [modeled after Ithaca](https://www.wbur.org/news/2018/05/09/guide-to-this-summers-boston-area-porchfests)'s event of the same name. Bands sign up to play music on their porches and people wander around the city listening. It was canceled in 2020, and postponed to the Fall in 2021, so this was the first one in two and a half years. (And I had missed the previous event because I was [travelling](https://www.jefftk.com/p/2019-tour).)

I had initially been hoping to play with [Kingfisher](https://www.kingfisherband.com/) and then [Whirlwind](https://cgi.neffa.org//public/showperf.pl?P=11204), But my bandmates weren't available and I ended up organizing a pickup group the day before. This ended up being super fun: unlike my regular bandmates, these were musicians I hadn't seen since before the pandemic (aside from my dad), and it was nice to be reminded of how much I like them all.

I've made a lot of changes to my [rhythm stage setup](https://www.jefftk.com/news/jammer) over the last few years, and it was really helpful to have a chance to run it live. You learn so much more than practicing at home: what does the crowd like? What sounds do you prefer in the moment? How can the interface better accommodate the demands of live performance? Afterwards I had a nice list of tweaks and new features to work on.

Partway through the set I noticed contradance folks informally dancing in our neighbors' empty driveway, and offered them a mic in case they wanted to call. A group of ~12 ended up dancing a few:

  
([youtube](https://youtu.be/LVz0gzNfD5U))

  
([youtube](https://youtu.be/DzEzGWW0P6k))

I feel somewhat conflicted on this. On one hand, I love playing for dancing and it was great to get the opportunity to do that for the first time in almost 2 years. I also heard from several of the dancers that they were really excited to get to dance. And it's probably not (much?) higher risk than what people are already doing.

On the other hand, it seems to me like this is a higher level of risk than our local contradance community has been comfortable with. The only organized dancing in Massachusetts I know of since before the pandemic was at Pinewoods this summer, with outdoor pavilions, and they [closed](https://www.pinewoods.org/pinewoods-camp-closing-early/) in mid-August, when cases were 2/3 of what they are now. Similarly, had been booked to play an outdoor contra in early September, which was cancelled in late August. Outdoor transmission remains very rare, however, and I'm not sure I would have cancelled an outdoor event.

(I used to organize [outdoor dances in Davis Square](https://www.jefftk.com/davis), but stopped when Lily was born. But it's probably too getting too cold to organize any this year?)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100230974367352)*
# Candy Innovation

What innovations in candy have there been since the '90s? Are there new flavors? Better imitations of existing flavors? New textures?

So far, all of the candy my kids have brought home seems to be things we could get 25 years ago. Though possibly flavors have improved, since I've only been trying what they've decided to share with me.

There have been some gains due to globalization, where candy that was previously hard to get in the US or unknown here is now more widely available, but has there been development beyond that?

(This post brought to you by yesterday's neighborhood, pinata)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100231014521882)*
# Andrew Yang on "The Priests of the Decline"

In this post excerpted from his new book, Andrew Yang uses his experience in politics to explain why the US political system rarely accomplishes much. It's strikingly similar to another recent post someone shared from Dominic Cummings: [Dominic Cummings : Regime Change #2: A plea to Silicon Valley - LessWrong](https://www.lesswrong.com/posts/aJg3An6P9ctEXw6dA/dominic-cummings-regime-change-2-a-plea-to-silicon-valley).

I've copied some of the key sections below:

"I call this dynamic constructive institutionalism — a tendency among leaders to state publicly and even hold the belief that everything will work out, despite quantitative evidence to the contrary, coupled with an inability to actually address a given institution’s real problems...

Indeed, two groups that are especially prone to constructive institutionalism are those that we rely upon both to give us a sense of the problems and to solve them — journalists and politicians.

Journalists are typically trained to be impartial observers, which inhibits them from expressing emotion or opinion. They are supposed to calmly document and present the news. For many, there is an implicit perch of authority and stability. Unfortunately, this has also turned many into market-friendly automatons and cultural guardians who make pro forma gestures about decorum, virtue, and propriety while ignoring the disintegration of trust, the dissipating integrity of their own organizations, or the decline of the American way of life...

If journalists are conditioned to calmly document dispassionately, politicians are conditioned to invoke profundity, resilience, and the greater good at every turn. As a politician you’re like a totem or shaman. You show up to a gathering or charity event to speechify and elevate the proceeding: “Thank you for the incredible work that you’re doing. It’s so important.” Which it is, of course. Though it would be if you didn’t show up too.

You are meant to embody the concerns of the community. You listen patiently to all. You are present. If someone asks you a question, you answer it reassuringly. You express values and aspirations. You are a human security blanket, and your job is to make people feel better."
# Open & Welcome Thread October 2021

(  
I saw October didn't have one. First post - please let me know if I do something wrong.  
To whoever comes after me: Yoav Ravid comments that the wording could use an update.  
)  
  
If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1). If you want to orient to the content on the site, you can also check out the new [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# How good is security for LessWrong and the Alignment Forum?

As far as I can tell, LessWrong/Alignment Forum haven’t been noticeably attacked or disrupted. However, I’m concerned that could change because:
- Attack command, control and automation will definitely improve, making more sophisticated attacks easier to deploy at scale. 
- If AI becomes as important as we think, and the Rationality community succeeds at  influencing its trajectory, LW/AF may become targets for surveillance/influence operations. 

So my questions are:
- How much focus does Lightcone Infrastructure and  put on security?
- Does Lightcone contract with any external security experts or penetration testers?
- Are the any plans to implement two factor authentication for LW/AF?
- Are there any planned responses if automated trolling/astroturfing attacks become much more common/advanced, as seems plausible with the rise of strong language models?
- Are there plans for secondary hosting providers, in case Amazon/the US become hostile?
- Is there some way we can download and backup all public conversations hosted on LW/AF?
- Relatedly, how are backups handled for LW/AF?

Thank you very much to Lightcone Infrastructure and the LessWrong team for your work. I’d be glad for any insight they (or anyone else) can share.
# 2021 Darwin Game - River

The River contains lots of every foragable food. The River is like the Human Garbage Dump except:
1. It has 5x as many resources.
2. Both air-breathing and water-breathing species can survive in the River.
3. Organisms may not start in the River. They must wander in.

![](https://s3.us-west-2.amazonaws.com/www.lsusr.com/lesswrong/darwin-game-2/river/river-1471191056N1l.jpg)

| Name | Carrion | Leaves | Grass | Seeds | Detritus | Coconuts | Algae | Lichen |
|---|---|---|---|---|---|---|---|---|
| River | 500 | 500 | 500 | 500 | 500 | 500 | 500 | 500 |

# Generation 0

At first, there are no animals and the foragables accumulate.

# Generations 1-100

![](https://s3.us-west-2.amazonaws.com/www.lsusr.com/lesswrong/darwin-game-2/river/river-0-100.png)

| Species | Original Biome | Venom? | Weapons | Armor | Speed | Forage? | Creator |
|---|---|---|---|---|---|---|---|
| Leafy Luncheoner | Temperate Forest | Neither | 0 | 0 | 1 | Leaves | Sean Hyer |
| Mooshroom | Grassland | Neither | 0 | 0 | 5 | Grass;Seeds | Vanessa |
| slow vulture | Desert | Antivenom only | 0 | 10 | 10 | Carrion | eat (Allows survival in the Desert),Corm |
| mediocre 1-2-s | Grassland | Neither | 1 | 0 | 2 | Seeds | Corm |
| Very Lesser Forest Dragon | Rainforest | Neither | 2 | 0 | 2 |  | Persephone 0461 |

The same pattern repeats for each foragable. First the foragable accomulates. Then a species which can consume the foragable appears. Its population explodes until it has eaten the accumulated foragable. Finally, its population reverts to carrying capacity.

Persephone 0461's Very Lesser Forest Dragon wandered in. It thrived until another predator appeared which hunted it to extinction.

# Generations 100-9000

Species appear. Species Disappear. Life is always in flux.

This is where all the Ocean's Soonbegons were coming from. Until they didn't anymore.

![](https://s3.us-west-2.amazonaws.com/www.lsusr.com/lesswrong/darwin-game-2/river/river-100-9000.png)

# Winners

| Name | Native Biome | Venom? | Attack | Armor | Speed | Forage? | Temperature Adaptation? | Creator | Social Media |
|---|---|---|---|---|---|---|---|---|---|---|
| Booyahs | Desert | Neither | 2 | 0 | 10 | Carrion | Heat (Allows survival in the Desert) | CK | None |
| Snark | Temperate Forest | Neither | 0 | 0 | 10 | Leaves |  | Vanessa | None |
| Venomoth | Rainforest | Venom + Antivenom | 0 | 0 | 3 | Grass |  | aphyer | None |
| Locust Eater Eater | Temperate Forest | Neither | 2 | 0 | 2 | Seeds |  | Multicore | None |
| Hopsplitter | Grassland | Neither | 0 | 0 | 10 | Grass;Seeds |  | Nem | None |
# Is GPT-3 already sample-efficient?

_(Concrete, easy-to-answer question below, explanation first)_

Common adage: Modern deep learning techniques are sample-inefficient; it takes loads of data for them to learn things. If you pre-train them, it takes less additional data for them to learn something new, but still compared to humans it takes a lot.

Elsewhere, based on papers like [this](https://arxiv.org/abs/2001.08361) and [this](https://www.lesswrong.com/posts/PStaXFAf6AzZdBm8u/scaling-laws-for-autoregressive-generative-modeling-henighan), various people have extrapolated the following takes:

--It seems like bigger neural nets need to see less data to reach the same level of performance.

--It seems like bigger neural nets need fewer epochs to reach convergence. Soon they'll only need to see each data point once. (Search [this](https://www.lesswrong.com/posts/k2SNji3jXaLGhBeYP/extrapolating-gpt-n-performance) for "multiple epochs")

I feel like this take is in tension with the common adage. **I wonder: If there is a fact mentioned in GPT-3's training data, how many times does it need to be mentioned before GPT-3 comes to know that fact?** For example, I'm told that GPT-3 knows the names of most prominent members of the rationalist community. How many times has it seen each name? Are we talking ten times, or ten thousand?*

I'd be interested to hear people do a bit of a **search for the "most sample-efficient/obscure fact"** in GPT-3's repertoire. In this manner we could quantity how many times GPT-3 needs to see something before it learns it. (Maybe we don't have access to the dataset used to train GPT-3. But people at Eleuther.ai have The Pile, right? And they've trained big transformers on it? We could answer the question easily and precisely there, no?)

Or am I thinking about this all wrong somehow? This seems like an obvious idea, I wonder why I haven't heard of it before.

*Suppose it is ten thousand. Then that means one in every ten million two-word strings on the internet is "Paul Christiano." (The dataset for GPT-3 was 300B tokens) Add in all the other rationalists/EAs and probably it means one in every hundred thousand words is the name of some prominent rationalist/EA. Surely this is too much, no? It seems way too much according to [Google Ngram Viewer](https://books.google.com/ngrams/graph?content=Paul+Christiano&year_start=1800&year_end=2019&corpus=26&smoothing=3&direct_url=t1%3B%2CPaul%20Christiano%3B%2Cc0#t1%3B%2CPaul%20Christiano%3B%2Cc0).
# Cheap food causes cooperative ethics

Nobody has ever fought a war over the oxygen in the air because there's plenty of oxygen to go around. We fight over things that are scarce and valuable. Historically, most wars have been fought over land and people. If you win you get to force your subjects to collect raw materials from the land. Sometimes these raw materials are mineral resources. Historically, most slaves have been forced to cultivate food.

It's hard to comprehend how important food staples used to be. In Edo Japan, wealth was measured in koku (石). One koku is (in theory) enough rice to feed one man for one year. The amount of koku a daimyo controlled was basically how many people he owned because a region's food staple production determined its carrying capacity and the human population grew until it hit carrying capacity. In other words, we bred until we were on the edge of starving to death. Most wars have ultimately been fought over land because land determines food production and food production was a matter of life and death.

My grandfather, who grew up in Taiwan before the green revolution, was too poor to afford rice. He ate sweet potatoes instead.

The green revolution of the 1950s and the 1960s increased food production faster than our population growth. On the Chinese version of TikTok[^1] there's a video of a guy eating a bowl of white rice with a spoonful of soy sauce. The comments are all of people feeling sorry for him. We have so much food these days that eating nothing but rice makes you surprisingly poor *even by Chinese standards*.

[^1]: 抖音 came first. Technically, TikTok is the American version of 抖音.

World War II ended in 1945—right before the green revolution. There has been no direct war between major world powers between then and now. I don't think this is a coincidence. Wars are usually about land and land is usually about food production and food production skyrocketed in the decades after World War II. (Birth control became widely adopted around the same time too.)

We still have small wars. We fight over oil and ideologies. But cars and capitalism aren't as important as food production. Fighting wars over food is stupid when food is cheap. Less fighting over food means less fighting overall. Countries being at peace with each other set the conditions for us to build more complex, interdependent trade networks. I think the idea that we're all human beings, regardless of our race, religion, sex and country of origin gained increasing power because cooperation is a winning strategy when there is enough to go around.

I predict that if *per capita* food production returns to the levels of ~~1914~~ 1776[^2] then so will humankind's ethics.

[^2]: Edit: See [comment](https://www.lesswrong.com/posts/cLehpZAY3HZuEQbHe/cheap-food-causes-cooperative-ethics?commentId=QoQsW3kKLS7eKgMEP) by Kaj_Sotala.
# Blog Post Day IV (Impromptu)

I decided to run another Blog Post Day. It's happening this Saturday the 9th. Sorry for the short notice!

Why do it? See [this post](https://www.lesswrong.com/posts/q972DHxCwZKFbEPMn/blog-post-day-unofficial).

If you are interested, you can join us online via this [special-purpose discord channel](https://discord.gg/j5TzpVF), or you can join us in person at the [CLR office lounge](https://longtermrisk.org/) in London. Email me if you are coming in person and I'll give you my phone number. Or you can use the discord.

The event will run from about 10am to about midnight. Come whenever and leave whenever. Mostly we'll just be on our computers writing, but at various points if desired we can take breaks and chat and discuss our planned posts etc.
# Glen Weyl: "Why I Was Wrong to Demonize Rationalism"

[Glen Weyl reflects](https://www.radicalxchange.org/media/blog/why-i-was-wrong-to-demonize-rationalism/) on his previous disparagement of the social scene surrounding this website, and expresses regret at having been too hostile: while he stands by many of his specific criticisms, he now thinks "rationalists" should be seen as more similar to other intellectual communities (which can be allied on some issues if not others), rather than a uniquely nefarious threat. (October 2021, 1300 words)
# Booster shots from same vs different vaccines

What is the current thinking on whether it's safe and effective to get a COVID booster shot from a different vaccine than the one you originally received? For example, getting a Pfizer shot if you originally got J&J. Is it *more* effective than getting a booster shot of the same type you originally got? References to good papers and analyses would be appreciated.