Harley-ml/lesswrong huggingface dataset:
# Optimism about Social Technology

In reaction to my morning dose of negativity from \[blogger's name redacted\], I am compelled to share some contrarian optimism:

I see a decent chance of significant improvements to the human condition in the near future due to breakthroughs in social technology. The historical precedent that comes to mind is the widespread adoption of insurance some hundreds of years ago. (Disclaimer: I skimmed the wikipedia article on insurance a while back and don’t really remember it all that well. Corrections welcome.) Prior to insurance contracts, your only option for hedging against catastrophes was to build up social capital with family & neighbors. Maybe people could expect more automatic community support back then, but this was at a time of drastically lower social and geographic mobility. I’m really glad that I live in a world with markets for explicit insurance policies (even if those policies and markets could be a lot better).

So I have to wonder if at least one of {dominant assurance contracts, quadratic funding, prediction markets, ring signatures, [attention auctions](https://www.lesswrong.com/posts/uA7rNDBBDo2yPNxw5/auctioning-off-the-top-slot-in-your-reading-list), retroactive public goods funding, [demand offsets](https://www.lesswrong.com/posts/qJxzBpubLB7Fr3oM8/demand-offsetting), smart contracts in general, the Prósperan legal system} might catch on and be as much of a game changer as formalized insurance contracts have been.

Sometimes I read someone and think “Well no wonder you’re always so pessimistic & sarcastic...for you, the phrase 'new technology' only evokes smartphone addiction, so you keep hanging all your hopes for the future on the vagaries of national politics.”

I'm motivated to say all this because I recently read a blog post about the collapse of legacy journalism (a hot topic in my echo chamber). The writer said that journalism has devolved into a monoculture of low-paid cultural elites, who are bound by reality very much less than they are bound by their social cliques. He's not wrong, but...he said it all with a such heavy note of despair in his voice. Meanwhile, I personally can’t help but imagine what crazy, cool, more incentive-aligned media model might rise from the ashes of the current system.
# How much interest would there be in a fringe theories wiki?

I've been exploring a concept for a wiki recently. The idea would be that people contribute fringe theories and present the best evidence for that theory, perhaps by contrasting it with mainstream interpretations of the data. Some examples of pages on the wiki could include,

*   [CellBioGuy's theory](https://www.lesswrong.com/posts/oLoLqmWYXgTyyiuXM/what-weird-beliefs-do-you-have?commentId=eFsdn6n9AHmuC6RrG) that the [paleocene-eocene thermal maximum](https://en.wikipedia.org/wiki/Paleocene%E2%80%93Eocene_Thermal_Maximum) was caused by an industrial civilization of birds living in Antarctica.
*   [Robin Hanson's theory](https://www.overcomingbias.com/2021/03/explaining-stylized-ufo-facts.html) that a panspermia sibling to Earth hosts an ancient advanced alien civilization, whose world government experienced complex system rot in the process of developing an Earth-monitoring system, resulting in bizarre UFO encounters with them that we sometimes hear about on Earth.
*   The theory from various cryonicists, following Eric Drexler, that future nanotechnology will be sufficient to repair damage from vitrification and subsequent cryopreservation.
*   Robin Gardiner's [theory](https://en.wikipedia.org/wiki/Titanic_conspiracy_theories#Olympic_exchange_hypothesis) that the ship called "The *Titanic*" was in fact the ship *Olympic,* and was purposely sunk as part of an elaborate insurance scam.
*   Scott Alexander's [pseudo-religious theory](https://slatestarcodex.com/2018/04/01/the-hour-i-first-believed/) that "There is an all-powerful, all-knowing logically necessary entity spawning all possible worlds and identical to the moral law."

The purpose would not be to make a determination to whether each theory was true or false, but rather just present the evidence.

Naturally, I'm more interested in theories that (1) have some sort of technical argument favoring it (regardless of credibility), (2) aren't merely moral or political theses in disguise, and (3) aren't already covered in sufficient detail in other places (unlike the JFK conspiracy theory). Wikipedia is a terrible place to do this, given their rules disallowing original research. Of course, without such constraints, there is a large risk that a Fringe Theories Wiki would attract bad editors, so some strict editing rules would still probably be required on the site.
# How will OpenAI + GitHub's Copilot affect programming?

[https://copilot.github.com/](https://copilot.github.com/)

[https://github.blog/2021-06-29-introducing-github-copilot-ai-pair-programmer/](https://github.blog/2021-06-29-introducing-github-copilot-ai-pair-programmer/)

> GitHub Copilot is powered by OpenAI Codex, a new AI system created by OpenAI. OpenAI Codex has broad knowledge of how people use code and is significantly more capable than GPT-3 in code generation, in part, because it was trained on a data set that includes a much larger concentration of public source code.

Will Copilot or similar systems become ubiquitous in the next few years? Will they increase the speed of software development or AI research? Will they change the skills necessary for software development?

Is this the first big commercial application of the techniques that produced GPT-3?

For anyone who's used Copilot, what was your experience like?
# How do the ivermectin meta-reviews come to so different conclusions?

[Ivermectin and outcomes from Covid-19 pneumonia: A systematic review and meta-analysis of randomized clinical trial studies](https://onlinelibrary.wiley.com/doi/10.1002/rmv.2265) comes to the conclusion: "Our study suggests that ivermectin may offer beneficial effects towards Covid-19 outcomes. More randomized clinical trial studies are still needed to confirm the results of our study."

On the other hand [Ivermectin for the treatment of COVID-19: A systematic review and meta-analysis of randomized controlled trials](https://academic.oup.com/cid/advance-article/doi/10.1093/cid/ciab591/6310839) comes to the conclusion: "In comparison to SOC or placebo, IVM did not reduce all-cause mortality, length of stay or viral clearance in RCTs in COVID-19 patients with mostly mild disease. IVM did not have an effect on AEs or severe AEs. IVM is not a viable option to treat COVID-19 patients."

What did the studies do differently to come to their conclusions? How do I go about interpreting which of them provides the better analysis?
# Is it true that most studies in Kahneman's 'Thinking fast and slow' are not replicable and the book is largely not supported by evidence?



So I stumbled on this article.

https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow


'So, replicability [of all studies in this book] is somewhere between 12% and 46%. Even if half of the results are replicable, we do not know which results are replicable and which ones are not.'


'Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologist, rather than an objective summary of scientific evidence.'


I have no background in social sciences or statistics so I don't know if claims and math in this article are correct.  Could somebody with more knowledge comment on this? This is HUGE if true. 


I love this book to pieces but I don't want to go around spreading outdated science.. are there any similar books on human biases and thinking which are more recent and more robust when it comes to evidence and statistics?
# Online LessWrong Community Weekend

Contact: [lesswrong.cw.2021@gmail.com](mailto:lesswrong.cw.2021@gmail.com)

A new team from Cologne and a couple other places decided to make the online LWCW happen again. We also got some of previous years' main organizers on board to help preserve the atmosphere of the event.

From 10th-12th September the Rationalist Community will again gather for 3 days of socializing, fun and intellectual exploration. There will be few pre-planned activities, but most of the content will be unconference style and participant driven.

On Monday, September 6th, participants will be invited to the event Discord and to access the collaborative schedule, to start planning what shall happen the following weekend. Over the week, the schedule will be filled in with workshops, talks and activities.

This is the 8th year of the LessWrong Community Weekend. Those of you who have attended this event before know that the atmosphere and sense of community at these weekends is something really special. If this sounds like something you would enjoy and you have some exciting ideas or skills to contribute, do come along and get involved.

This year the event will happen online for the second time. We will use a different Discord server, but you can expect the same atmosphere and rhythm to the event.

**Applications**
----------------

Apply [here](https://docs.google.com/forms/d/e/1FAIpQLSdA94s73C0fFHz6jGDJ1mroQp559sMSjOj06bqurSKdij6zpA/viewform?usp=sf_link) before **September 1st**.

**Some Platforms we will use**
------------------------------

The list of platforms is identical to last year’s one.

(Don’t worry if the list seems long. Most of these work in your browser, so all you have to do is to follow the link that will be provided for that activity.)

*   [Discord](https://discord.com/) will be used as the main communications hub for the event. This is where you will go to find all other information. Activities that don’t happen in Discord will be linked to from Discord.
*   [Teamup](https://teamup.com/) calendar for collaborative scheduling of activities. Each activity will also have, if needed, a link to the platform hosting it.
*   [Zoom](https://zoom.us/) and/or [Google Meet](https://meet.google.com/). None of the above platforms are great for really large gatherings such as the opening and closing sessions and probably some of the more popular talks. For these things we will use Zoom and/or Google Meet. Which one depends on the number of participants and the needs of these participants.
*   [Gatheround](https://gatheround.com/) (previously Icebreaker) will be used for speed-friending sessions.
*   Possibly more…? There are likely to be activities that require some specific platform, e.g. online board games. But activity organizers are discouraged to use anything that requires installation. The most probable exception is [Tabletop Simulator](https://www.tabletopsimulator.com/).
# A (somewhat beta) site for embedding betting odds in your writing

I've made [a site for publishing predictions and publicly offering to bet on them](https://biatob.com). LessWrong seems like a place that might be interested in such a tool, and I expect at least 4 people will be interested enough in this site to play around with it a little[![(bet: $20 at 50%)](https://biatob.com/p/14075895152523535559/embed-lesswrong-14pt.png)](https://biatob.com/p/14075895152523535559). It's a little rough around the edges, but I'd be delighted if my target audience (you) found it intriguing enough to try it / offer feedback!

---

**What's it do, more specifically?** Well, there are two basic flows:
- *Offer bets:* Alice has some belief that other people might disagree with! She operationalizes it into some statement like "by Sept 1, I will have 2k karma on LessWrong"; she figures she's 60% sure, and willing to lose up to $50. She creates a prediction on Biatob with those parameters, then publicizes that prediction (e.g. by pasting a little HTML snippet into a LW post).
- *Take bets:* Bob reads Alice's LW post and sees the little "[bet: $50 at 60%+]" link. He thinks her goal is unrealistically high, but he respects Alice and trusts her to pay up if she's wrong. Bob clicks the link, signs up, and bets $20 against her $30.

Then, on Sept 1, Alice gets an email reminding her to check whether the prediction came true or not. When she does, Biatob tells her how much money [she/Bob] owes [Bob/her], and they settle up offline. (Notice that this is all honor-system: you can only bet against people you trust, but at that cost, you don't have to trust my janky web site with your money.)


**For example:** I signed up, created a couple of predictions (like the above one about a afew LWers signing up), and copied some HTML into this post. Now I can say things like "I doubt anybody will even be interested enough to make an easy $10 by betting against me.[![(bet: $10 at 50%)](https://biatob.com/p/3929815011859091110/embed-lesswrong-14pt.png)](https://biatob.com/p/3929815011859091110)" (That prediction is a bit tongue-in-cheek, but I'm completely serious that I'll pay up if you go to the trouble of signing up to bet against me. I'm hoping to get some user feedback from you.)
# How would you run the statistics on whether Ivermectin helped India reduce COVID-19 cases?

I frequently read the claim that India reduced their COVID-19 cases a lot via Ivermectin but it's very unclear to me whether the relevant data to make that claim is cheripicked. 

India has 34 region and each of those region has their own policy on Ivermectin. While we don't have exact data about Ivermectin usage, Google Trend data provides a proxy for usage of Ivermectin and does provide data that's fine-grained enough to separate individual region. John Hopkins provides data for COVID-19 cases by Indian region as well. 34 regions sounds to me like it's plausible to find statistically significant results if Ivermectin had an effect that's as great as proponent of Ivermectin claim

Given p-hacking concerns, it makes sense to first decide on how to analyse the data and then run the statistics. Does anyone have suggestion for how the data should be analysed?
# Open and Welcome Thread – July 2021

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1). If you want to orient to the content on the site, you can also check out the new [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# You are allowed to edit Wikipedia

I frequently hear complains from people about individual Wikipedia pages but most of the people who complain only complain outside of Wikipedia. Wikipedia is inherently democratic. If you read a Wikipedia article and think it's very problematic, take five minutes and write about why it's problematic on the talk page of the article. 

Wikipedia is an important part of the commons. If you think from an EA perspective those five minutes (or even more if it takes you time to search for sources) have a good chance of being time spent with a good EA return. 

While recruiting people outside of Wikipedia to individual pages to engage in discussion goes against Wikipedia's rules, simply engaging on Wikipedia and voicing your opinion is helpful. It makes it more likely that consensus on the article shifts in the right direction.
# Musing on the Many Worlds Hypothesis

The All, it seems, cannot commit  
But at each crossroads makes a split.  
When quanta have a chance to vary,  
It answers “both” to every query,  
Nine tredecillion times a minute  
At every place with quanta in it.  
With each mitosis, it is reckoned,  
One universe creates a second  
That from the other slips away  
(Least that’s what the equations say).  
  
If what the physics says is true  
I have my doppelgangers too:  
Each minute tredecillion nine  
For each quantum I call mine.  
Each then spawns another clutch,  
Which strikes me as a me too much.  
From each seed mote a dynasty  
Emerges from me like a tree.  
Like pollen I from there disperse  
In countless destinies diverse.  
  
What if I play Russian Roulette?  
I prune the tree; it survives yet.  
And this leads me to suppose  
The “me” survivors will be those  
(Especially as time advances)  
Who survived by uncanny chances.  
In one of these unnumbered threads  
Each coin I flip will come up heads.  
In some set (infinite, though small)  
Schrödinger’s Cat outlives us all.  
  
We all beat odds to be alive  
(a third of embryos survive).  
Fell down a cliff; got just a sprain,  
A roof tile dropped and missed my brain,  
Suicidal adolescence,  
Internal organs in senescence,  
Passing out while on a bender:  
Look at that: just bent a fender.  
In all this how was death denied?  
Unlikely? No: I also died.  
  
But as I tend my future me,  
How ought I care for this vast tree  
That sprouts fathomless avalanches  
Of egos in its greedy branches?  
Could I, would I make the call:  
“All for one, and one for all!”?  
For now let’s zoom in here below,  
To the only me I’m sure I know.  
My tire is flat, I’m out of glue.  
And I’ve forgotten what to do.
# Preschooler Turnovers

I recently got into making proper flaky pastry, and while it's a bit of a hassle it's really very tasty. I think I started with [this recipe](https://www.seriouseats.com/old-fashioned-flaky-pie-dough-recipe), and I've been getting decent at it:

[![](https://www.jefftk.com/flaky-turnovers.jpg)](https://www.jefftk.com/flaky-turnovers-big.jpg)

The kids really like the turnovers, and will ask for them much more often than I'm up for.

A few days ago Anna (5y) really wanted turnovers for breakfast. I told her I didn't want to do the work, but if she really wanted them she could cook them herself. She said she did. Instead of trying to walk her through a finicky butter recipe, worrying about keeping the butter solid and making sure she didn't overwork the flour, I decided to adapt a simple oil-based one:

*   1C flour
*   1/4C oil
*   2T milk

I set out the ingredients already measured. At some point I'll probably show her how to do that part too.

You mix everything together with your hands, nice and squishy. It doesn't much matter how, the goal is to just get it homogenous. Nothing to be careful with.

Once it's combined, you shape it into a ball, and plop it down onto a baking sheet. Press it flat with your hands, then roll it:

[![](https://www.jefftk.com/rolling-out-oil-crust.jpg)](https://www.jefftk.com/rolling-out-oil-crust-big.jpg)

You put some chocolate chips in the middle, and fold it closed. Optionally, brush on some egg yolk. Anna likes to brush on excessive amounts, it's lots of fun:

[![](https://www.jefftk.com/turnover-egg-wash.jpg)](https://www.jefftk.com/turnover-egg-wash-big.jpg)

Bake at 400F until browned, about 12min. (I always set a timer to check on things for a few minutes before the recipe says, since you can always cook something a bit longer.)

[![](https://www.jefftk.com/oil-turnover-finished.jpg)](https://www.jefftk.com/oil-turnover-finished-big.jpg)

It is not a very high-quality pastry, but everything tastes better when you do it yourself, and that counts for a lot, especially when you're five.

[![](https://www.jefftk.com/eating-oil-crust-turnover.jpg)](https://www.jefftk.com/eating-oil-crust-turnover-big.jpg)

(For a vegan version, substitute soy milk. You can leave off the egg wash, or substitute Just Egg.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100222517350282)*
# Rationality Yellow Belt Test Questions?

Since joining the Official Idealized Rationality Organization as a fresh novice six months ago, you have been training and studying hard, and today you are ready to take your exam to advance to the first rank of recognized skill and knowledge. 

You expect the test to be difficult. Here in the Future, card-carrying Rationalists are widely known to be formidably effective, clear-thinking people. Rationalists are capable of impressive feats individually, and accomplish miracles when working in groups. Part of cultivating such a strong reputation, obviously, involves setting high standards.

You step out of your flying car, enter the testing center, and take your seat. The proctor hands you a thick sheaf of test questions. Turning the first page, you read the first question ...

*(What sorts of question would you hope to see on such a test? If not "test questions" per se, what other sorts of requirements would make sense?)*

*(Another way of looking at this might be: design a test that you would be proud to say you passed.)*
# Intelligence without Consciousness

I sleep-talk. In fact, I don't just talk in my sleep, I can carry on a conversation. Advice and insight I have is much lower quality than when I'm awake, but I'm mostly coherent. Once someone recorded a conversation with me, and it was pretty funny. For example: "A lot of people like Batman. We should buy stock in Batman".

I'm not conscious, though. I'm not self-aware. If I have qualia, it's a very tiny amount of qualia. I definitely wouldn't wonder "do I have qualia?" If I didn't keep waking up, there wouldn't be a moral reason for others not to treat me how they find most convenient.

What does this mean? It means that [Nonsentient Optimizers](https://www.lesswrong.com/posts/HsRFQTAySAx8xbXEc/nonsentient-optimizers) and [Nonperson Predicates](https://www.lesswrong.com/posts/wqDRRx9RqwKLzWt7R/nonperson-predicates) are definitely possible. There is some threshold of intelligence and/or self-awareness needed to change a neural net into a person.

(Lucid dreaming is a separate issue. I'm talking about something that happens even when not lucid dreaming.)

Also, people in the comments keep saying maybe I did have consciousness and just don't remember it. No: the conversation quality is sort of like [this](https://www.cnet.com/news/man-records-sleep-talking-for-a-year-adam-rosenberg-the-somniloquist/). When people talk to me, my response is usually connected to what they said, but I'm definitely not conscious.
# Announcing My Free Online Course "Original Seeing With a Focus On Life"

In May and June of 2021, I ran an online nature study course with about two dozen participants. I had three goals:

1. Help people create or deepen a personal connection with nature.
2. Learn how to run month-long online courses with over a dozen participants.
3. Test some hunches about the rationality material I’ve been working on.

One of my main hunches was that if you complete this course, you’ll find you’re able to study your own mind, or just about anything else, in the way a naturalist studies nature, with no further guidance. I obviously need more data, but the preliminary feedback looks promising.

[A solo version of the course is now available for free through my website](https://www.loganstrohl.com/nature-study), thanks largely to support from the [Long Term Future Fund](https://funds.effectivealtruism.org/funds/far-future). 

If you want to get better at original seeing, I know of no better resource. I hope that some of you will try it out and tell me how it goes.
# Anthropics in infinite universes

When talking about anthropics, people often say things like "assume the universe is finite; weird things happen in infinite universes". I've myself [argued](https://www.lesswrong.com/posts/99isjR8W4JGZdro7x/sia-fears-expected-infinity) that [SSA](https://www.lesswrong.com/tag/self-sampling-assumption) breaks down when we encounter infinities; [SIA](https://www.lesswrong.com/tag/self-indication-assumption) breaks down sooner, when we encounter expected infinities.

![](https://www.dropbox.com/s/x593drtlc739qsq/Iexist.png?raw=1)

You can formalise this informally[^yes] with the thought that:

[^yes]: Yes, it's perfectly possible to formalise informally, and I encourage people to do it more often.

1. In an infinite universe, anything can happen, no matter how unlikely: life must exist somewhere. So our existence doesn't tell us anything about life; its probability could be anything at all.

A superficially convincing argument; but not one you'd use for anything else. For instance, consider the following:

2. In an infinite universe, anything can happen, no matter how unlikely: if gravity didn't exist, somewhere it must seem to exist by shear chance. So our observation of gravity doesn't tell us anything about gravity; its probability could be anything at all.

I've argued before that [anthropic questions are pretty normal](https://www.lesswrong.com/posts/3kwwDieE9SmFoXz9F/non-poisonous-cake-anthropic-updates-are-normal). Why would we accept the reasoning in question 1, but reject it in question 2?

We shouldn't. We can deal with questions like 2 by talking about limits of probabilities in larger and larger spaces, or by discounting distant observations (similar to sections 2.3 and 3.1 in [infinite ethica](https://www.nickbostrom.com/ethics/infinite.pdf)). So we might define conditional probabilities like $P(X \mid Y)$ in an infinite universe in the following way:

* Let $P_{rl}(X \mid Y)$ be the ratio of observers, within a large hypersphere of radius $r$ centered on location $l$, that observe $X$ and $Y$, relative to the proportion that observes $Y$. If this tends to a limit as $r\to \infty$, independently of $l$, then define that limit to be $P(X \mid Y)$.

Note that this definition works just as well for $Y=$ "we observe the force of gravity to be blah" as with $Y=$ "we exist".

Now, that definition might not be ideal (in particular, "radius" is not defined for relativistic space-time). No problem: different definitions of probability are asking different questions, and can lead to different anthropic probabilities, [just as in the finite case](https://www.lesswrong.com/posts/LARmKTbpAkEYeG43u/anthropics-different-probabilities-different-questions).

I'll call these class of questions "SIA-limit questions", since they are phrased as ratios of observers, and dependent on how we use limits to define probability in infinite universes. They each lead to various "SIA-limit anthropic probability theories"; in most standard situations, these should reach the same answers as each other.
# Practical anthropics summary

This is a simple summary of how to "do anthropics":

* If there are no issues of exact copies, or advanced decision theory, and the questions you're asking aren't weird, then use SIA. And by "use SIA", I mean "ignore the [definition of SIA](https://www.lesswrong.com/tag/self-indication-assumption), and just do a conventional Bayesian update on your own existence". Infinite universes won't be a problem (any more than they are with conventional probabilities). And this might not increase your expected population by much.

First of all, there was the realisation that different theories of anthropic probability correspond to [correct answers to different questions](https://www.lesswrong.com/posts/LARmKTbpAkEYeG43u/anthropics-different-probabilities-different-questions
) - questions that are equivalent in non-anthropic situations.

We can also directly answer "what actions should we do?", without talking about probability. This [anthropic decision theory](https://www.lesswrong.com/s/kmryZRz5r9bjsug9e) gave behaviours that seem to correspond to SIA (for total utilitarianism) or SSA (for average utilitarianism).

My personal judgement, however, is that the SIA-questions are more natural than the SSA-questions (ratios of totals rather than average of ratios), including the decision theory situation (total utilitarianism rather than average utilitarianism). Thus, in typical situations, using SIA is generally the way to go.

And if we ignore exact duplicates, Boltzmann brains, and simulation arguments, SIA is simply [standard Bayesian updating on our existence](https://www.lesswrong.com/posts/BYy62ib5tAkn9rsKn/sia-is-basically-just-bayesian-updating-on-existence). Anthropic probabilities can be computed exactly the same way as [non-anthropic probabilities can](https://www.lesswrong.com/posts/3kwwDieE9SmFoXz9F/non-poisonous-cake-anthropic-updates-are-normal).

And there are fewer problems than you might suspect. This [doesn't lead to problems with infinite universes](https://www.lesswrong.com/posts/rbJLrcmHtusGBudTY/anthropics-in-infinite-universes) - at least, no more than standard probability theories do. And anthropic updates tend to increase the probability of larger populations in the universe, [but that effect can be surprisingly small](https://www.lesswrong.com/posts/xfEsxAtBTLgFe7fSZ/the-sia-population-update-can-be-surprisingly-small) - $7$ to $32$ given the data we have.

Finally, note that anthropic effects are [generally much weaker](https://www.lesswrong.com/posts/wgHbNZHsqfiXiqofd/anthropics-and-fermi-grabby-visible-zoo-keeping-and-early) than Fermi observation effects. The fact that we don't see life, on so many planets, tells us a lot more than the fact we see life on this one.
# BASALT: A Benchmark for Learning from Human Feedback

Copying the abstract of the [paper](https://arxiv.org/abs/2107.01969):

> The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are now being incorporated into commercial products. With this comes an additional challenge: how can we build AI systems that solve tasks where there is not a crisp, well-defined specification? While multiple solutions have been proposed, in this competition we focus on one in particular: learning from human feedback. Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve.
> 
> The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. These tasks are defined by a paragraph of natural language: for example, "create a waterfall and take a scenic picture of it", with additional clarifying details. Participants must train a separate agent for each task, using any method they want. Agents are then evaluated by humans who have read the task description. To help participants get started, we provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline that leverages these demonstrations.
> 
> Our hope is that this competition will improve our ability to build AI systems that do what their designers intend them to do, even when the intent cannot be easily formalized. Besides allowing AI to solve more tasks, this can also enable more effective regulation of AI systems, as well as making progress on the value alignment problem.

I also mention this in the [latest Alignment Newsletter](https://www.alignmentforum.org/posts/a7YgzDYx4FhdB3TmR/an-155-a-minecraft-benchmark-for-algorithms-that-learn), but I think this is probably one of the best ways to get started on AI alignment from the empirical ML perspective: it will (hopefully) give you a sense of what it is like to work with algorithms that learn from human feedback, in a more realistic setting than Atari / MuJoCo, while still not requiring a huge amount of background or industry-level compute budgets.

Section 1.1 of the paper goes into more detail about the pathways to impact. At a high level, the story is that better algorithms for learning from human feedback will improve our ability to build AI systems that do what their designers intend them to do. This is straightforwardly improving on [intent alignment](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment) (though it is not solving it), which in turn allows us to better govern our AI systems by enabling regulations like "your AI systems must be trained to do X" without requiring a mathematical formalization of X.
# For reducing caffeine dependence, does daily maximum matter more, or does total daily intake matter more?

I'm trying to reduce my dependence on caffeine from needing ~160mg in the morning to needing 80mg. My strategy is to slowly reduce my intake, dropping maybe 20mg every 3 days. So far so good. 

But sometimes I feel tired later in the day, and also want caffeine. So I have 80mg more, 10 hours after the morning 160mg. Is this going to screw up my attempt at tolerance reduction? 

The argument for Yes: Daily caffeine intake is what matters. You can't reduce yourself down to needing less than 160mg in the morning by taking 240mg a day.

The argument for No: The important thing is the max amount of caffeine in you at any given time. Since the half-life of caffeine is between [1.5 and 9.5 hours (with a middle guess of 5 hours)](https://www.ncbi.nlm.nih.gov/books/NBK223808/), by evening, you only have 160mg * 0.5 * 0.5 = 40mg in your system. Adding 80mg brings you to 120mg. This is less than the 160mg max dose your body is used to hitting, so it's not a big deal. As long as you keep the maximum caffeine in your system below 160mg, you can keep reducing your tolerance.

Which of these arguments is more right? Or is there a third view of the situation, better than either?
# Anthropic decision theory for self-locating beliefs

This is a link post to the "[Anthropic decision theory for self-locating beliefs](https://arxiv.org/pdf/1110.6437.pdf)" paper, with the abstract:

>This paper sets out to resolve how agents ought to act in the Sleeping Beauty problem and various related anthropic (self-locating belief) problems, not through the calculation of anthropic probabilities, but through finding the correct decision to make. It creates an anthropic decision theory (ADT) that decides these problems from a small set of principles. By doing so, it demonstrates that the attitude of agents with regards to each other (selfish or altruistic) changes the decisions they reach, and that it is very important to take this into account. To illustrate ADT, it is then applied to two major anthropic problems and paradoxes, the Presumptuous Philosopher and Doomsday problems, thus resolving some issues about the probability of human extinction.

The key points of that paper are also available in [this post sequence](https://www.lesswrong.com/s/kmryZRz5r9bjsug9e).
# What would it look like if it looked like AGI was very near?

I am writing a post on scenario-based planning, and thought it would be useful to get some example scenario symptoms on the topic of near-term AGI. 

So the scenario-based planning thought exercise goes like this: 

Imagine it is ten years from now and AGI has happened (for the thought experiment just imagine you are looking at history from the outside and are capable of writing this even if all humans were turned into paperclips or whatever) and you are writing a post on how in retrospect it was obvious that AGI was just around the corner.

What symptoms of this do you see now? Contingent on this scenario being true, what symptoms would you expect in the next 10 years? 

Bonus points if you write it from the point of view of your fictional self 10 years from now who has experianced this scenero.
# [Link] Musk's non-missing mood

Luke Muehlhauser writes:

> Over the years, my colleagues and I have spoken to many machine learning researchers who, perhaps after some discussion and argument, claim to think there’s a moderate chance — a 5%, or 15%, or even a 40% chance — that AI systems will destroy human civilization in the next few decades. [^1^](https://lukemuehlhauser.com/musks-non-missing-mood/#footnote_0_6351) However, I often detect what Bryan Caplan has called a “[missing mood](https://www.econlib.org/archives/2016/01/the_invisible_t.html)“; a mood they would predictably exhibit if they really thought such a dire future was plausible, but which they don’t seem to exhibit. In many cases, the researcher who claims to think that medium-term existential catastrophe from AI is plausible doesn’t seem too upset or worried or sad about it, and doesn’t seem to be taking any specific actions as a result.
> 
> Not so with Elon Musk. Consider his reaction ([here](https://www.youtube.com/watch?v=ycPr5-27vSI&t=1218s) and [here](https://www.youtube.com/watch?v=ycPr5-27vSI&t=1870s)) when podcaster Joe Rogan asks about his AI doomsaying. Musk stares at the table, and takes a deep breath. He looks *sad*. Dejected. Fatalistic. Then he says:

[Read more](https://lukemuehlhauser.com/musks-non-missing-mood/)
# What will the twenties look like if AGI is 30 years away?

I'm especially interested to hear from people with long (i.e. 20+ year) timelines what they think the next 10 years will look like.

\[EDIT: After [pushback from Richard Ngo](https://www.lesswrong.com/posts/axbTNGuMtB4hCkNus/what-will-the-twenties-look-like-if-agi-is-30-years-away?commentId=MTF2AAufpJhcCxpPX#KMYERswLHeETfEGnt), I've agreed to stop talking about short and long timelines and just use numbers instead, e.g. "20+ year" and "<10 year" timelines. I recommend everyone do the same going forward.\]

Ajeya (as of the time of [her report](https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit) at least) is such a person, and she gave some partial answers already:

> By 2025, I think we could easily have decent AI personal assistants that autonomously compose emails and handle scheduling and routine shopping based on users’ desires, great AI copy-editors who help a lot with grammar and phrasing and even a little bit with argument construction or correctness, AIs who summarize bodies of information like conversations in meetings or textbooks and novels, AI customer service reps and telemarketers, great AI medical diagnosis, okay AI counseling/therapy, AI coding assistants who write simple code for humans and catch lots of subtle bugs not catchable by compilers / type checkers, even better AI assisted search that feels like asking a human research assistant to find relevant information for you, pretty good AI tutors, AIs that handle elements of logistics and route planning, AIs that increasingly handle short-timescale trading at hedge funds, AIs that help find good hyperparameter settings and temperature settings for training and sampling from other AIs, and so on.

But she thinks it'll probably take till around 2050 for us to get transformative AI, and (I think?) AGI as well.

The hypothesis I'm testing here is that people with long timelines \[EDIT: 20+ year timelines\] nevertheless think that there'll be lots of crazy exciting AI progress in the twenties, just nothing super dangerous or transformative. I'd like to get a clearer sense of how many people agree and what they think that progress will look like.

(This question is a complement to [Bjartur's previous question](https://www.lesswrong.com/posts/YRtzpJHhoFWxbjCso/what-would-it-look-like-if-it-looked-like-agi-was-very-near))
# CGP Grey on the difficulty of knowing what's true [audio]

Starts 00:34:40, ends 00:49:06. Features discussion of tracking down sources for historical 'facts' to discover that the sources don't say what people say they say, the difficulty of knowing what's true, and exasperation at the idea that companies such as YouTube should take down videos with false things in them. No transcript available, unfortunately.
# Essentialness of Data

In discussing online privacy, people will sometimes say things like " [if the data isn't required to operate, you don't need it](https://news.ycombinator.com/item?id=27824695)." If I can turn on tracking protection, stop sending data, and nothing breaks, then clearly it wasn't needed, right? But consider these three cases:

*   Learning from implicit feedback: dictation software can operate without learning what corrections people make, or a search engine can operate without learning what links people click on, but the overall quality will be lower. Each individual piece of information isn't required, but the feedback loop allows building a substantially better product.
    
*   Incremental rollouts: when you make changes to software that operates in complex environments it can be very difficult to ensure that it operates correctly through testing alone. Incremental rollouts, with telemetry to verify that there are no regressions or that relevant bugs have been fixed, produces better software. Even Firefox [collects telemetry by default](https://support.mozilla.org/en-US/kb/telemetry-clientid).
    
*   Ads: most websites are able to offer their writing for free, without a paywall, because they can get paid for showing ads. Collecting more data makes ads more efficient, which makes them more profitable for the sites, which translates into more competition to provide users with things to read. ([more](https://www.jefftk.com/p/why-i-work-on-ads))
    

Instead of pushing for "don't collect data", I think it would make a lot more sense for advocates to push for "only collect data privately" and work to make that easier (carrot) or mandatory (stick). None of these uses require individual level data, they're just easiest to implement by sending all of the data back to a central server and processing it there.

(What does "private" mean? Ideally it means that no one reviewing the data can tell what your, or any other individual's, contribution was. This is formalized as [differential privacy](https://en.wikipedia.org/wiki/Differential_privacy), and is typically implemented by adding noise proportional to the maximum contribution any individual could have. In some cases [k-anonymity](https://en.wikipedia.org/wiki/K-anonymity) may also provide good protection, but it's [trickier](https://www.jefftk.com/p/kanon). And this is only the beginning; privacy researchers and engineers have been putting a lot of work into this space.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100223236094912)*
# Lafayette: traffic vessels

This week I’m in Lafayette, a town merely twenty-three minutes further from my San Franciscan office than my usual San Franciscan home, thanks to light rail. There are deer in the street and woods on the walk from the train to town.

On this occasion at least, Lafayette doesn’t feel properly like a small town to me. I think it’s the main road. A lot of the town is spread along this single road, but the road itself doesn’t feel like its main deal is being Lafayette’s main road. It feels more focused on being an energetic transitway between somewhere and somewhere else, neither in Lafayette. Which probably isn’t even that true, since there is a perfectly giant highway also spanning Lafayette just North of it. Maybe the problem is that it’s too wide, so that the town feels like it’s tenuously accumulated on the sides of a road like plaque, rather than the road being an organic vessel of the town. Or, you know, I’m imagining things.

I seem to imagine things a lot regarding some kind of road Feng Shui (note: I know nothing about actual Feng Shui). My mind natively reads roads as conduits of some kind of ‘energy’, and tries to apply appropriate intuitive physics. For instance, if you have big flows in and out of a place, relative to the place itself, it won’t feel like its own place. It will feel like a section of a larger place. For instance, the typical random intersection in a big American city can’t be a _place_ with its own local vibe, where you might feel like staying, because it can’t be very separate from the surrounding city that its energy (traffic?) is constantly being exchanged with. It’s just going to feel like a section of various energetic routes.

This intuitive physics is sort of like the physics of streams with leaves and debris in them. For a place to be placelike, and appealing to settle down in, it needs to have enough nooks or ponds or complications for the fast flowing streams in and out to eddy around in and slow down and let the debris be still. And this main street is a big stream running through a small place.

This is all contradicted by the frequency with which people stand in narrow thoroughfares at parties even in the face of literal physical streams of partygoers pressing against them. (My intuition on this case is that the pressure of the partygoer liquid is so high that it somehow makes sense to be stuck in the doorway, but I don’t explicitly see how this model even makes sense.) I don’t know of any pro evidence for this model, but my brain just keeps on having it.
# Lafayette: empty traffic signals

Seeking to cross a road on the walk into downtown Lafayette, instead of the normal pedestrian crossing situation, we met a button with a sign, ‘Push button to turn on warning lights’. I wondered, if I pressed it, would it then be my turn to cross? Or would there just be some warning lights? What was the difference? Do traffic buttons normally do something other than change the lights? I clearly believe they do. They make it my turn. But they don’t send a wire into the ‘whose turn is it’ variable deep in the ought-sphere, so what exactly do they do?

I suspected that this button didn’t change whose turn it was, and it felt empty and devoid of some special potence of being a traffic button.

I liked to imagine that it was just a normal traffic button, but taking a more nihilistic view of its role. In which case, its nihilistic view seemed to have practical consequences! It wasn’t being as good a good traffic button while saying that it didn’t change whose turn it is. It genuinely fails to coordinate the traffic so well, because here am I unable to garner the ‘right’ to cross with confidence, and there are the drivers unsure what I’m doing. But shouldn’t a traffic button be able to do its job regardless of its philosophical commitments, or without pretending to have philosophical commitments it doesn’t have?

One might say that the thing going on is that it being ‘my turn’ is a fact about everyone’s expectations. For instance, if the drivers will expect me to cross, then it is ‘my turn’. (I’m tempted to say ‘if the drivers think it is my turn, then it is my turn’, but what are the drivers then thinking?) This doesn’t seem quite right, in that the drivers themselves are asking whether this light means that it is the pedestrian’s turn, and all of us seem to be asking something about the underlying truth, not about each other. Also, if I run erratically into the road, the drivers and I may both come to expect that I am going to cross, but it still isn’t my turn.

I fantasized that I had woken up in a new world which was just like the old world, but where everything was like the traffic light. I would phone the doctor’s office later to ask if it was ok to cancel my appointment this late, they would just say, ‘I’ll change what it says in this schedule’.

‘But is it _ok_?’

‘I will not record your cancellation.’

‘Should I pay you?’

‘I am not charging you’

‘But is that like a favor, or is it the policy? Have I wronged your medical practice? Do I owe you _really_? Tell me if I was bad!’

‘I erased your name from this box on my piece of paper.’

My tentative take is that turns are real, and we created them, and traffic buttons have genuine power over them, and if a traffic button doesn’t recognize their existence it is potentially at a real disadvantage, perhaps in a similar way to how a chair maker who doesn’t recognize chairs as a thing is at a disadvantage.

(To be clear, I expect philosophers have much better thought out views on this, and welcome people telling me what they are–this is me thinking aloud, not philosophical advice.)