Harley-ml/lesswrong huggingface dataset:
# Interest in a digital LW "book" club?

Prompted by [interest](https://forum.effectivealtruism.org/posts/qtDZHBx6LpAR7ejDD/what-standalone-lesswrong-posts-would-you-recommend-to-most) in LW on EA forum, I thought people might be interested in a digital book club. We'd pick a couple classic posts every 2-4 weeks. One option is a group zoom meeting or slack, but we could also just invite people to read and comment on the selected articles. I'd be happy to come up with some guiding questions and pick articles if a handful of people are interested.
# Prediction Markets are for Outcomes Beyond Our Control

Betting markets are the gold standard of expert predictions because bets are the ultimate test of what people truly believe. 

The best betting markets are highly liquid. A liquid market is one where you can place a large bet without moving the price very much. Liquid prediction markets work when no individual person can influence the outcome. Bettering markets are a great way to find out if "it will rain tomorrow" or whether "candidate $x$ will be elected president next year".

But what if a single person can influence the outcome? For example, what would happen if I created a betting market for "Lsusr will publish a blog post tomorrow"?

Suppose I am ambivalent about whether I will publish a blog post tomorrow. If the price of "Lsusr will publish a blog post tomorrow" drops below 1.00 then I will buy shares of "Lsusr will publish a blog post tomorrow" and then pocket a risk-free profit by posting a blog post tomorrow. If the price of "Lsusr will publish a blog post tomorrow rises above 0.00 then I will buy shares of "Lsusr will not publish a blog post tomorrow" and then pocket a risk-free profit by not posting a blog post tomorrow.

The market equilibrium occurs even if I am unaware that the prediction market exists. Suppose the price drops to 0.99. A trader could buy shares and then pay me a small fee to influence the outcome.

I am not truly ambivalent. Suppose I'm willing to influence the outcome in exchange for $500. What happens? If the market liquidity is less than $500 then we have a functional prediction market. If the market liquidity is more than $500 then we have a regular market.

Prediction markets function best when liquidity is high, but they break completely if the liquidity exceeds the price of influencing the outcome. Prediction markets function only in situations where outcomes are expensive to influence.
# Why rationalists are not much concerned about mortality?

As of 2022, humans have a life expectancy of ~80 years and a hard limit of ~120. Most rationalists I know agree that dying is a bad thing and at minimum we should have an option to live considerably longer and free of the "diseases of the age", if not indefinitely. It seems to me that this is *exactly* the kind of problem where rationality skills like "taking things seriously", "seeing with fresh eyes" and awareness of time discounting and status quo bias should help one to notice something is very very wrong and take action. Yet - with the exception of cryonics[^1n9byve5mu2] and a few occasional posts on LW - this topic is largely ignored in the rationality community, with relatively few people doing the available interventions on the personal level, and almost nobody actively working on solving the problem for everyone.

I am genuinely confused, why is this happening? How is it possible that so many people who are equipped with epistemological tools to understand they and everyone they love are going to die, understand it's totally horrible, understand this problem is solvable in principle, can keep on doing nothing about it?

There is a number of potential answers to this question I can think of, but none of them is satisfying and I'm not posting them to avoid priming.

\[ETA: to be clear, I have spent a reasonable amount of time and effort making sure that the premise of the question is indeed the case - whether rationalists are insufficiently concerned about mortality - and my answer is unequivocal "yes". In case you have evidence to the contrary, please feel free to post them as an answer\]

[^1n9byve5mu2]: It's an interesting question exactly how likely cryonics is to work and I'm planning to publish my analysis of this at some point. But unless you assign a ridiculously optimistic probability to it working, the problem largely remains. Even 80% probability of success would mean your chances are worse than in Russian roulette! Besides, my impression is that only a minority of rationalists is signed up anyway.
# EleutherAI's GPT-NeoX-20B release

> GPT-NeoX-20B is a 20 billion parameter autoregressive language model whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights. In this paper, we describe the model architecture and training, evaluate its performance, and discuss the broader impacts of its release. We are open-sourcing the training and evaluation code, as well as the model weights, at [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).

The [alignment section](http://eaidata.bmk.sh/data/GPT_NeoX_20B.pdf#page=11) of the paper would probably be especially of interest to this community.
# [Linkpost] TrojanNet: Embedding Hidden Trojan Horse Models in Neural Networks

[TrojanNet: Embedding Hidden Trojan Horse Models in Neural Networks](https://ar5iv.org/html/2002.10078?fbclid=IwAR0R7Brl3R6SLbRqFetSDM6hH0nV83duJHFYTFUkyIin6-1FWI9zZK9yxiM)

TL;DR: 

> TrojanNet embeds a neural network within another neural network, and it's NP-hard to detect - which limits transparency.

Abstract:

> The complexity of large-scale neural networks can lead to poor understanding of their internal details. We show that this opaqueness provides an opportunity for adversaries to embed unintended functionalities into the network in the form of Trojan horses. Our novel framework hides the existence of a Trojan network with arbitrary desired functionality within a benign transport network. We prove theoretically that the Trojan network’s detection is computationally infeasible and demonstrate empirically that the transport network does not compromise its disguise. Our paper exposes an important, previously unknown loophole that could potentially undermine the security and trustworthiness of machine learning.

(found [on FB](https://www.facebook.com/nellwatson/posts/10160036197108559) and apparently suggested by old-timer [XiXiDu](https://www.lesswrong.com/users/xixidu))
# A Nice Representation of the Laplacian

_Epistemic status: some proofs here may well contain a small bug._

The Laplacian of a function from n-dimensional space to the real line is a generalization of the second derivative of a normal function: specifically, it’s the sum of second derivatives of the function along each dimension. One day, I recall coming across a theorem that said that the Laplacian of a function was equal to something like the limit of the average function value on a shell around a point, minus the function value at that point, divided by the squared radius of the shell. This is a nice theorem, because it connects the Laplacian to one of my favourite mathematical objects, the graph Laplacian. However, I later couldn’t find this theorem anywhere. Yesterday, I decided to re-derive the theorem, and today I decided to write it up. You can see the theorem and its proof [here](https://danielfilan.com/pdfs/laplacian_representation.pdf). Do let me know if there are any errors I should fix.
# Epsilon is not a probability, it's a cop-out

A lot of people, after learning on this site that 0 and 1 are "not probabilities," (they are, just ask a mathematician, but they are not useful for Bayesian updating) quickly switch to using "epsilon" or "one minus epsilon" instead, meaning "a reeeeaaaalllly tiny number, but not technically zero". My interpretation of this is lazy signaling of "I am a good Bayesian" without actually doing the work. Why? Because if you ask this person what kind of evidence would change their estimate from "epsilon" to, say, 5%, they would be hard pressed to come up with anything sensible. 

Well, that's the descriptive part, which is much easier than the prescriptive part: what to do instead of just labeling what you consider a negligible probability epsilon and never updating it. As an aside, the complexity difference between identifying issues and successfully fixing them is generally like P vs. NP. 

Probably in some cases one can actually put a number on a tiny probability (e.g. the odds of the sun rising tomorrow, without any additional data, is about 1-1/average number of consecutive sunrises). In other cases it would be more accurate to say "I have no idea, I would be super surprised if the Pascal's mugger was telling the truth, but who knows, I have been super surprised before." In yet other cases what one deems "epsilon" would be something logically inconsistent (like successfully two-boxing in a perfect predictor Newcomb's setup). Surely there are other cases, as well. But I think it pays thinking through what exactly your "epsilon" means in a given situation.
# Scott Alexander Visit and Mixer

**NOTE:** Location has been updated as of 5 pm CT, February 19. 

The Austin LW/SSC group will be hosting Scott Alexander of AstralCodexTen ([announcement](https://astralcodexten.substack.com/p/austin-meetup-next-sunday) on his blog) on Sunday, February 27th. See event details for the location.   
  
We will provide lunch (breakfast tacos including vegetarian and vegan options), snacks, and drinks (including water).
  
Scott will only be there until 3, but we have the place until 8 pm.

**Parking:** The parking lot at Moontower will fill up quickly, but you can park on the street around the venue.
# SPARC Applications Are Now Live!

The Summer Program for Applied Rationality and Cognition (SPARC) is a camp intended for really bright high school students to get together and become more “agenty.”  It was originally spun up by people around CFAR to get math students interested in AI alignment and EA, though the focus has now expanded to include more than that. I attended SPARC last year, and it was sufficiently incredible that I want to bring the news to any high schoolers here that read Lesswrong.  

Please apply – You won’t regret it!

As somebody who was and still is a frequent reader of LessWrong, it might not be at first apparent that getting to interact with others your age and talking about many of the same ideas would be so pivotal.  But your monkey brain will disagree.  In theory, you know that agenty people exist, who work on interesting things, and that you could be one of them, but until you actually meet them, it doesn’t become real.  Furthermore, finding your “tribe” of young individuals like yourself is not something to be overlooked, especially if you don’t live in a major rationalist hub. Finding people you click with is genuinely amazing.  Anyway, I don’t want to go on for too long, but even if this sounds even marginally appealing, please do actually sit down and writing yourself a reminder to apply.  Be the agent you want to see in the world :) 

**How do I apply?**

[**https://www.sparc-camp.com/apply**](https://www.sparc-camp.com/apply)

*Note: The program is free for all participants, with financial assistance for travel available for those who need it.*

**Applications are due March 6th.  **

More firsthand accounts of SPARC can be found here:

*   [https://www.sparc-camp.com/](https://www.sparc-camp.com/)
*   [https://www.quora.com/What-benefits-do-students-get-from-attending-the-SPARC-summer-program](https://www.quora.com/What-benefits-do-students-get-from-attending-the-SPARC-summer-program) 
*   [https://www.quora.com/What-is-it-like-to-attend-SPARC-1](https://www.quora.com/What-is-it-like-to-attend-SPARC-1)
# The Big Picture Of Alignment (Talk Part 1)

I recently gave a two-part talk on the big picture of alignment, as I see it. The talk is not-at-all polished, but contains a lot of stuff for which I don't currently know of any good writeup. Major pieces in part one:

*   Some semitechnical intuition-building for high-dimensional problem-spaces.
    *   Optimization compresses information "by default"
    *   Resources and "instrumental convergence" without any explicit reference to agents
*   A frame for thinking about the alignment problem which only talks about high-dimensional problem-spaces, without reference to AI per se.
    *   The central challenge is to get enough bits-of-information about human values to narrow down a search-space to solutions compatible with human values.
    *   Details like whether an AI is a singleton, tool AI, multipolar, oracle, etc are mostly irrelevant.
*   Fermi estimate: just how complex are human values?
*   Coherence arguments, presented the way I think they *should* be done.
    *   Also subagents!

Note that I don't talk about timelines or takeoff scenarios; this talk is just about the technical problem of alignment.

Here's the video for part one: 

Big thanks to Rob Miles for editing! Also, the video includes some good questions and discussion from Adam Shimi, Alex Flint, and Rob Miles.
# Do any AI alignment orgs hire remotely?

tl;dr: qualified software engineer considering what their next job might be; now thinking about direct work as a serious option.

Previous plan was something like:

    SELECT * FROM big_tech_co
    WHERE location = 'remote'
    ORDER BY team_fit, salary
    

For a variety of reasons, I'm not a huge fan of this plan anymore.

New plan:

1.  Check the job pages of all the AI alignment orgs I know
2.  Check 80000 Hours jobs board in case I missed something
3.  ???
4.  Post question on LW

I didn't find anything looking at the job pages of the AI alignment orgs that I'm familiar with, and 80000 Hours didn't bring up anything that fit the bill either, so here we are.

Me:

*   mission-aligned, long-time member of the community (since ~2013) and meetup organizer (since 2017)
*   fairly strong software engineer, mostly backend-focused but have recently picked up enough React to be able to meaningfully contribute to an existing project on the front-end (no design skills to speak of, yet). Can also rapidly onboard myself to an unfamiliar codebase. Legible artifacts demonstrating these claims beyond a resume include:
    *   a [non-trivial contribution to major open-source library](https://github.com/grpc/grpc-node/pull/1952)
    *   going from zero to one with a [SaaS product](https://protocall.dev) on the side while working full-time
*   happy to travel something like 10% of the time, especially to the Bay, to integrate professionally & socially
*   happy to do something like a part-time work trial (outside of my core working hours, but including weekends), and willing to take a few days off to do this in-person if the fit seems good
*   in general, happy to do some non-standard things that might not be expected of me at a typical tech job, be agent-y, etc., particularly to compensate for the downsides to an organization of my working remotely

You:

*   an organization that works either directly on AI alignment, or a "meta" org that e.g. better enables others to work on AI alignment
*   willing to hire someone on these (or similar) terms!
*   most everything else I'm looking for seems pretty strongly correlated with that, and I'm aware that there aren't exactly a wealth of options to filter on

Does anyone know of any orgs that I might have missed?

*   Lightcone & Redwood don't seem to be hiring remotely
*   Alignment AI doesn't specify whether they're hiring remotely (but since they're in the UK, you'd imagine they'd say if they were)
*   MIRI & ARC aren't hiring engineers

Most of the other orgs I'm familiar with seem to be doing differently-targeted work (i.e. Ought), or are doing work which seems to boil down to "capabilities advancement", but I'm open to arguments here if I've misjudged one or more of them.
# What's the status of TDCS for improving intelligence?

I recently heard about transcranial direct current stimulation as a technology for improving memory and attention, and I thought it sounded like a great area to attempt some munchkin activity.

However, it seems like the science isn't very clear.

Does anyone have a more informed perspective on the efficacy behind it?
# Transformer inductive biases & RASP

This is a paper by Weiss et al. about the "**Restricted Access Sequence Processing Language**", which is a computational framework for what transformers can do.

| DL architecture | Computational framework |
| --- | --- |
| Feedforward networks | Circuits |
| RNNs | Finite state machines |
| Transformers | RASP (this paper) |

Any program written in RASP can be implemented straightforwardly by a transformer.  Indeed, the authors demonstrate that RASP can be "compiled" to a transformer, and that transformers can be trained to "mimic a RASP solution".

From the abstract (emphasis mine):

"We map the basic components of a transformer-encoder—attention and feed-forward computation—into simple primitives, around which we form a programming language: the Restricted Access Sequence Processing Language (RASP). We show how RASP can be used to program solutions to tasks that could conceivably be learned by a Transformer, and **how a Transformer can be trained to mimic a RASP solution**. In particular, we provide RASP programs for histograms, sorting, and Dyck-languages. **We further use our model to relate their difficulty in terms of the number of required layers and attention heads: analyzing a RASP program implies a maximum number of heads and layers necessary to encode a task in a transformer.** Finally, we see how insights gained from our abstraction might be used to explain phenomena seen in recent works."

One implication of this is that program length (simplicity) in RASP may be a good approximation to the inductive bias of transformers.  I will probably use this in future work along the lines of "[How complex are myopic imitators?](https://www.alignmentforum.org/posts/2eRgFFeeS7pR4R8nD/how-complex-are-myopic-imitators-1)".

**If you know of any tricks that transformers use that are not covered by RASP, please leave a comment -- it would be highly useful to me.**
# What's the easiest way to currently generate images with machine learning?

As far as I understand, the images for the latest LessWrong books were generated by a neural net. I would like to generate images for the sequences I have on LessWrong in a similar way. What's the easiest way for me to do so?
# The Big Picture Of Alignment (Talk Part 2)

I recently gave a two-part talk on the big picture of alignment, as I see it. The talk is not-at-all polished, but contains a lot of stuff for which I don't currently know of any good writeup. Linkpost for the first part is [here](https://www.lesswrong.com/posts/xdSDFQs4aC5GrdHNZ/the-big-picture-of-alignment-talk-part-1); this linkpost is for the second part. 

Compared to the first part, the second part has less material which has not been written up already, although it does do a better job tying it all into the bigger picture than any already-written source. I will link to relevant posts in the outline below.

Major pieces in part two:

*   [Programs as a compressed representation for large (potentially infinite) probabilistic causal models with symmetry](https://www.lesswrong.com/posts/Xd9FLs4geRAWxkQPE/writing-causal-models-like-we-write-programs)
    *   Potentially allows models of worlds larger than the data structure representing the model, including models of worlds in which the model itself is embedded.
    *   Can't brute-force evaluate the whole model; must be a lazy data structure with efficient methods for inference
*   [The Pointers Problem](https://www.lesswrong.com/posts/gQY6LrTWJNkTv8YJR/the-pointers-problem-human-values-are-a-function-of-humans): the inputs to human values are latent variables in humans' world models
    *   This is IMO the single most important barrier to alignment
*   Other aspects of the "type signature of human values" problem (just a quick list of things which I'm not really the right person to talk about)
*   Abstraction (a.k.a. ontology identification)
    *   [Three](https://www.lesswrong.com/posts/jJf4FrfiQdDGg7uco/the-telephone-theorem-information-at-a-distance-is-mediated) [roughly-equivalent](https://www.lesswrong.com/posts/vvEebH5jEvxnJEvBC/abstractions-as-redundant-information) [models](https://www.lesswrong.com/posts/FWuByzM9T5qq2PF2n/a-correspondence-theorem) of natural abstraction
*   Summary (around 1:30:00 in video)

I ended up rushing a bit on the earlier parts, in order to go into detail on abstraction. That was optimal for the group I was presenting to at the time I presented, but probably not for most people reading this. Sorry.

Here's the video:

Again, big thanks to Rob Miles for editing! (Note that the video had some issues - don't worry, the part where the camera goes bonkers and adjusts the brightness up and down repeatedly does not go on for very long.) The video includes some good questions and discussion from Adam Shimi, Alex Flint, and Rob Miles.
# Is Anyone Else Seeking to Help Community Members in Ukraine Who Make Refugee Claims?

Some rationalists and effective altruists in British Columbia have begun mulling over the option of whether they would be able to sponsor a fellow community member living in Ukraine to stay as a refugee in Canada. There is [a process](https://www.canada.ca/en/immigration-refugees-citizenship/corporate/publications-manuals/guide-private-sponsorship-refugees-program/section-2.html?fbclid=IwAR0FA0q5ZDPiZpMrYQ9-5XO_ENj4HWKsgaOPWksTucs-hjbhH5m4K9mdsPw) by which 5 or more citizens of Canada can sponsor a refugee. Have you or anyone else you know in the community begun considered doing something similar or other ways to aid peers still in Ukraine?
# You have a place to stay in Sweden, should you need it.

If anyone here is in trouble of any kind, and finds themselves in Sweden with no place to stay, or in need of other help, I want to help you. Send me an email at [johan.domeij@gmail.com](mailto:johan.domeij@gmail.com) for further contact.

I am posting this now due to the ongoing conflict in Ukraine, but the offer is not limited to that conflict. I am not able to give significant financial support, but I have a car and a place to stay for several people.
# Preparing for Possible Russian Countersanctions in Europe

Europe is about to introduce stronger economic sanctions against Russia. Given Putin's past behavior I think it is likely that there will be Russian countersanctions.

The most relevant could be cutting off of gas for Europe. In the long run this   
would be self defeating because it accelerates energy independence from Russia.  
But nevertheless I think this is quite possible (est. prob: 25%). If this would come at the same time as below average temperatures in March and April in countries heavily depending on natural gas for heating (e.g. in Germany > 50% if you include district heating), then many households could find themselves temporarily without heating (est. prob: 10%). Of course, that is nothing compared to the hardship of the Ukrainian people but nevertheless it could make sense to make some cheap preparations:

1\. If your heating is based on gas or on gas based district heating: Buying an electric  
fan heater. Cheap devices are currently sold on Amazon for less than 20 Euros. (Yesterday, I have paid 16 Euros for a heater).

2\. If your cooking stove is based on gas and you don't have an electric alternative   
(e.g. a microwave oven): Buying an electric hot plate. Single hot plates are currently about 20 Euros.

Both purchases would have to be made before Putin announces a gas embargo because, I believe, after that they will be sold out immediately.

Anything else on could do to prepare?
# [David Chapman] Resisting or embracing meta-rationality

Excerpting the intro. Follow the link for the rest.

> [Meta-rationalism](https://metarationality.com/rationalism-definitions#meta) may provoke skepticism or hostility at first. It contradicts fundamental [rationalist](https://metarationality.com/rationalism-definitions#rationalism) principles that you may hold explicitly, or that you may have absorbed implicitly, from technical education and the culture of technical work.
> 
> Alternatively, [meta-rationality](https://metarationality.com/rationalism-definitions#meta) may seem intriguing, if implausible and incomprehensible. Then you may ask:
> 
> *   Is there really anything to this?
> *   If so, what kinds of problems can I solve with it that I couldn’t deal with rationally? (I bet there aren’t any!)
> *   What principles of rationality does meta-rationality contradict? (Rational principles are absolute truths, so I don’t see how meta-rationality could have better ones.)
> *   How can I learn the supposed meta-rational methods, if they’re worth anything?
> 
> These questions are natural and rational. They also miss the point, and therefore make it impossible to understand what meta-rationality is:
> 
> *   You are already doing meta-rationality all the time. You can’t do rationality without it. “What there is” to meta-rationalism is discovering that you mostly do meta-rationality mindlessly, and therefore ineptly. Then you can learn to do it better.
> *   Meta-rationality isn’t about solving problems; that’s what rationality is for. Meta-rationality can help you use rationality more effectively. By understanding problems and potential solution approaches in broader contexts, you become more likely to solve them—but you’ll do that rationally. You’ll also get better at deciding which problems are worth solving, and effective ways of dealing with problems other than “solving” them.
> *   Meta-rationality doesn’t run on principles. Demanding that it does makes it impossible to learn.
> *   Meta-rationality also doesn’t have methods.
> 
> Meta-rationality is a *different kind of thing* from rationality, so you can’t learn it the same way. You learn rationality by getting principles and methods from lectures or textbooks, and then working through toy exercise problems. That won’t work for meta-rationality.
# Late 2021 MIRI Conversations: AMA / Discussion

With the release of [Rohin Shah and Eliezer Yudkowsky's conversation](https://www.lesswrong.com/posts/tcCxPLBrEXdxN5HCQ/shah-and-yudkowsky-on-alignment-plans), the **Late 2021 MIRI Conversations** sequence is now complete.

This post is intended as a generalized comment section for discussing the whole sequence, now that it's finished. Feel free to:

*   raise any topics that seem relevant
*   signal-boost particular excerpts or comments that deserve more attention
*   direct questions to participants

In particular, Eliezer Yudkowsky, Richard Ngo, Paul Christiano, Nate Soares, and Rohin Shah expressed active interest in receiving follow-up questions here. The Schelling time when they're likeliest to be answering questions is **Wednesday March 2**, though they may participate on other days too.
# What are some low-cognitive -workload tasks that can help improve the world?

I have a chronic health condition which results in significant brain fog, usually lasting a few hours most days. When I don’t have brain fog I can work on high-level technical work with relative ease, but when I’m experiencing symptoms, even simple tasks become a challenge (and manual labor is out of the question). As a result, most activities I’d like to do in the Effective Altruism space are out of reach for a good portion of my day. What are some ways to put that time to optimal use?

Beyond myself, it would be nice to have a list of low-cognitive-workload altruistic tasks to refer people to if they don’t have the intellectual capability to directly work on the alignment problem or whatever, but still want to help make the world a better place.
# Are our community grouphouses typically rented, or owned?

Asking partially because the Auckland community really needs to start a grouphouse and could do with advice, and also I expect the answers to be interesting for a bunch of other reasons:

*   Intuitively, one might expect a mostly longtermist community to be interested in buying, rather than renting, and if they did, that could end up being quite interesting in economic and social terms, different residents may end up owning potentially different-sized 'shares' of the property, selling them when they leave (to the house, or to the new resident?) and buying them when they join. All of these things would be somewhat socially challenging, full of lessons, and worth talking about. It might also provide motivation to work on a less ambitious form of my [Propinquity Cities](https://www.lesswrong.com/posts/hTwshHAce4qZHL5zE/propinquity-cities-so-far), which can pretty much be paraphrased as a matchmaking system/placement solver for grouphouses.
*   Alternately (and this is the impression I get), maybe everyone just rents. It would be the path of least resistance and you can make some arguments about the benefits of dynamism. In which case I'd want to start a discussion of whether that really makes sense, and whether it has made sense so far.
*   Alternately, maybe one person owns the house, and everyone else pays rent to them?
# March 2022 Welcome & Open Thread

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the new [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# The Agent

Before the war I didn't really think about anything. I just did things and things just happened to me. Then yesterday my mother asked me if we should leave. "Before it's too late". That was the first time I ever really thought about something. This is what thinking was like for me:

\- I pictured all the animals and trees and hay bales and garbage and driftwood and streams and bogs and winds and bridges and messages and tunnels between us and freedom.

\- I pictured every horrible thing they would do to us.

\- I pictured firing a gun.

\- I pictured being shot with a gun.

\- I pictured being buried under rubble.

\- I pictured moving rubble off of my mother.

\- I pictured reading a book from my childhood in a refugee camp.

\- I pictured my mother so tired that I would carry her.

\- I pictured us committing suicide together.

\- I pictured learning a new language.

\- I pictured the stars without smoke.

Then I balanced each of these on a giant scale, inside a calculator, which assigned numbers to each of these, and multiplied each by how likely they were, and calculated how good it would be to go, and how good it would be to stay.

It was much better to go. I told my mother so. She didn't ask how I decided. She packed us food, and off we went, into the heavy heat of the night, pausing only to listen to the whistling of the bombs.
# Would (myopic) general public good producers significantly accelerate the development of AGI?

This question is material to us, as [we're building an impact certificate market](https://www.impactcerts.com/) (a major component in [retroactive public goods funding](https://astralcodexten.substack.com/p/lewis-carroll-invented-retroactive?s=r)), and if the answer is yes, we might actually want to abort, or — more likely — I'd want to put a lot of work into helping to sure up mechanisms for [making it sensitive to long-term negative externalities](https://forum.effectivealtruism.org/posts/eDwcke3TbbZKAYkgi/being-an-individual-alignment-grantmaker?commentId=vArciPg3BeN2eN7dg).

Another phrasing: **Are there any dependencies for AGI, that private/academic AI/AGI projects are failing to coordinate to produce, that near-future foundations for developing free software would produce?**

I first arrived at this question with my economist hat on, and the answer was "of course, there would be", because knowledge and software infrastructure are non-excludable goods (useful to many but not profitable to release). But then my collaborators suggested that I take the economist hat off and try to remember what's actually happening in reality, in which, oh yeah, **it genuinely seems like all of the open source code and software infrastructures and knowledge required for AI are being produced and freely released by private actors, in which case, us promoting public goods markets couldn't make things worse**. (Sub-question: Why is that happening?)

But it's possible that that's not actually happening, it could be a streetlight effect: Maybe I've only come to *think* that all of the progress is being publicly released because I don't see all of the stuff that isn't! Maybe there are a whole lot of coordination problems going on in the background that are holding back progress, maybe OpenAI and Deepmind, the algorithmic traders, DJI, and defense researchers are all doing a lot of huge stuff but it's not being shared and fitted together, but a lot of it would be in the public cauldron if an impact cert market existed. I wouldn't know! Can we rule it out?

It would be really great to hear from anyone working on AI, AGI, and alignment on this. When you're working in an engineering field, you know what the missing pieces are, you know where people are failing to coordinate, you probably already know whether there's a lot of crucial work that no individual player has an incentive to do.
# The Geometric Series of 1/(d+1) is a Fraction in Base-d

Suppose $x$ is a positive number less than 1. What is the sum of the positive powers of $x$?

For example, suppose $x=\frac12$.

$$\begin{eqnarray}
\sum_{i=1}^\infty x^i&=&\sum_{i=1}^\infty \frac1{2^i}\\
&=&\frac1{2^1}+\frac1{2^2}+\frac1{2^3}+\ldots\\
&=&\frac12+\frac14+\frac18+\ldots\\
&=&1
\end{eqnarray}$$

The case where $x=\frac12$ is intuitively obvious in base-10, but it's even more intuitively obvious in base-2. I will use a subscript to indicate base *e.g.* $0.5=0.1_2$, $4=100_2$, $16=10000_2$ and $10_{17}=17$.

$$\begin{eqnarray}
\sum_{i=1}^\infty x^i&=&\sum_{i=1}^\infty \frac1{10_2^i}\\
&=&\sum_{i=1}^\infty 0.1_2^i\\
&=&0.1_2+0.01_2+0.001_2+0.0001_2+\ldots\\
&=&0.111111111_2\ldots\\
&=&0.\bar 1_2\\
&=&1
\end{eqnarray}$$

The above trick works for the inverse of any positive integer. Suppose $x=\frac1{17}$.

$$\begin{eqnarray}
\sum_{i=1}^\infty x^i&=&\sum_{i=1}^\infty \frac1{10_{17}^i}\\
&=&\sum_{i=1}^\infty 0.1_{17}^i\\
&=&0.1_{17}+0.01_{17}+0.001_{17}+0.0001_{17}+\ldots\\
&=&0.111111111_{17}\ldots\\
&=&0.\bar1_{17}\\
&=&\frac1{16}\\
\end{eqnarray}$$

We can generalize to any denominator $d$.

$$\begin{eqnarray}
\sum_{i=1}^\infty \frac1{d^i}&=&\frac1{d-1}\\
\sum_{i=0}^\infty \frac1{d^i}&=&1+\frac1{d-1}\\
&=&\frac{d-1}{d-1}+\frac1{d-1}\\
&=&\frac d{d-1}\\
&=&\frac 1{1-\frac1d}\\
\end{eqnarray}$$

The relationship holds even when $d$ is not an integer. Let $r=\frac1d$.

$$\sum_{i=0}^\infty r^i=\frac1{1-r}$$

This is the equation for a geometric series.
# A Warm Up Problem for Yudkowsky's New Fiction

Hello, this is my first post on lesswrong, a community I deeply value. Feel free to tell me to RTFM, especially if you can point me there.

Yesterday I attended the weekly rationalist meetup at "The Territory" in Seattle for the first time. I attended for many reasons, but I was particularly excited to discuss Yudkowsky's new novel with others. His novel can be found here: [https://www.glowfic.com/posts/4582](https://www.glowfic.com/posts/4582)

Only one of the eleven or so people there had heard about it, so this post is my attempt to market the book. Since all marketing should also add value, I will now move on to adding value by giving you a warm-up problem for some of the math and concepts you will encounter in this novel.

The problem is as such:

> You are playing a game against ROB, the robber. In this game, ROB will choose two distinct arbitrary (real/computable) numbers, A and B. ROB will send both numbers to TABI the arbiter, who will secretly flip a fair coin and then hand you either A or B depending on the outcome of the flip. Your task is to come up with an algorithm which does better than 50% accuracy in determining if you were handed the larger number.

I will post the solution tomorrow, and then I will highlight specific ways I have applied the lessons from Yudkowsky's latest work in real/professional life.

Note that the book will not explicitly spoil this problem for you, and vice versa.

Edit: solution is [here.](https://www.lesswrong.com/posts/s57EudtibFHyMGEqM/deterministic-strategies-can-be-suboptimal)
# PSA: if you are in Russia, probably move out ASAP

The Metaculus crowd thinks that it's very likely that Russia will restrict emigration:

I myself am more optimistic about this question resolving negatively. But even if Russia would not close borders for a notable fraction of its citizens, it seems fairly likely that the aviation industry will collapse or shrink due to sanctions. 

Related forecasts:  
\- [https://www.metaculus.com/questions/9918/martial-law-in-russia-before-april-2022/](https://www.metaculus.com/questions/9918/martial-law-in-russia-before-april-2022/)    
\- [https://manifold.markets/DanielFilan/will-martial-law-be-declared-in-rus](https://manifold.markets/DanielFilan/will-martial-law-be-declared-in-rus)
# Higher Risk of Nuclear War

With the Russian invasion of Ukraine, nuclear war is still very unlikely, but I'd guess it's the most likely it's been in my lifetime. It's worth thinking through how this impacts any [disaster plans](https://www.jefftk.com/p/disasters) you might have:

*   What would you do [if you got an incoming ICBM alert](https://www.jefftk.com/p/what-if-we-got-that-alert)? Are there supplies you would wish you had that are worth sorting out now?
    
*   If you live in a major city or near a likely military target, is there a level of escalation at which you would want to leave for somewhere less populous? Is there planning for that which would make sense to do now?
    
*   If war elsewhere causes serious supply chain disruptions, is there anything you depend on (ex: [medication](https://www.jefftk.com/p/emergency-prescription-medication)) that it's worth trying to get extra of now?
    

(This post started [as a comment](https://www.facebook.com/jefftk/posts/10100133222353172?comment_id=10100245002754352).)
# [Letter] Russians are Welcome in America

Edit [to clarify]: American *immigration policy* is unwelcoming to immigrants. Often, the only ways to enter the US legally are to marry someone in the US or to get the sponsorship of a powerful tech company. It takes several months even under the best of circumstances. I just wanted clarify that Russians *living in the US* face no prejudice.

---

Dear [redacted],

I'm sorry. I had notifications turned off and didn't see your message until right now.

Get out of Russia before the borders close! If your in-group is LW-adjacent techies then you're probably smart enough to find a great job in the USA that pays way more than you could ever earn in Russia (even before the war and the sanctions).

> I ask you for advice on how to help them find safe place in the foreign social fabric? Is there anyone left who can offer them student\IT positions w/t being ostracised?

Lol.

Ostracism is a complete non-issue for Russians in the United States. (And, presumably, other Western nations.) I have never met a single person in real life with hatred towards Russians. Nobody hates Russians here. Russians fit right in. All you need is good English (and, judging by your message, your English is fine (even if it's not, English isn't too hard for Russian-speakers to learn)).

Don't get me wrong. The United States has problems with racism, prejudice and xenophobia. But it's not directed towards Russians living in America. Getting a visa is hard. But that's it. Once you're in, you're in. (Don't worry about obtaining an American visa right now. Just get out of the country immediately. Worry about obtaining an American visa later.)

You know that [Russian rocket scientist friend I was advertising](https://www.lesswrong.com/posts/QMB5c5aypEWLxSAjQ/ukraine-situation-report-2022-03-01#_Advertisement__Want_to_hire_a_Russian_rocket_scientist_to_write_your_software_)? I have already gotten contacts from three companies that want to help him acquire employment and a visa. And not just three ordinary companies. Three of the best places to work in the entire United States of America. I can't promise you the same, but I hope my words help reassure you. We literally have [a giant statue](https://en.wikipedia.org/wiki/Statue_of_Liberty) in New York symbolizing how welcoming this country is to immigrants.

Your priorities should be.

1. Get the hell out of Russia before the borders close.
2. Secure your short-term finances.
3. Start figuring out how to get a tech job.

Sincerely,

Lsusr
# Don't Let Personal Domains Expire

It's common to see advice along these lines:

> Don't build your stuff in someone else's sandbox. Get your own domain and point it to whatever service you choose to use. Your email address should be @yourdomain, where you have full control and no one can lock you out. Don't fall into the trap of digital sharecropping.

There are complicated tradeoffs here and different choices will make sense for different people, but it's close to what I do personally. My writing and projects are hosted on my own domain \[1\] and my email is `jeff@jefftk.com`.

On the other hand, I don't think this is something to do lightly. Say you register `you.example` and start going by `you@you.example`. A few years later you decide this is too much hassle, switch to using `you@fastmail.com` or `you@gmail.com`, and let `you.example` expire. Someone else can register it, send email legitimately as `you@you.example`, and [Angular gets compromised](https://thehackerblog.com/zero-days-without-incident-compromising-angular-via-expired-npm-publisher-email-domains-7kZplW4x/). If there is anywhere you forgot to remove your former email from your profile, now you are open to being impersonated.

This problem isn't unique to personal domains, but it's much more likely: the major email services don't make abandoned email addresses open to reregistration, to avoid exactly this issue.

If you're considering registering a domain to use as your online identity, make sure you're willing to take on the cost and hassle of keeping the domain registered indefinitely.

  
\[1\] I do cross-post to Facebook, LessWrong, and occasionally other places. I also rely on them to host discussions on my posts, though I attempt to archive those discussions back on my site.
# Accelerated [Honors] Calculus

The University of Washington offers the following introductory math classes:
- **MATH 111 Algebra with Applications** is for people who dislike math.
- **MATH 124 Calculus with Analytic Geometry I** is for people who like math.
- **MATH 134 Accelerated [Honors] Calculus** is for people in an abusive relationship with math.

I took Accelerated [Honors] Calculus.

Accelerated [Honors] Calculus starts with the Peano axioms. The Peano axioms are a set of nine assertions like "0 is a natural number", "$x=x$" and "if $x=y$ the $y=x$". It is possible to derive all of number theory using just the Peano axioms. Add the Cauchy Completeness Axiom and you can derive calculus too.[^1]

[^1]: My memories of the class are hazy. Please correct me if I'm wrong about which axioms are necessary to prove what.

Our assignment was to to derive calculus.

I tried out CSE 143: Computer Programming II too. CSE 143 used Java. The class was based around [memorizing words](https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password) like "polymorphism" and "postorderal traversal". My friend majoring in Human Centered Design and Engineering felt like CSE 143 was hard. I felt like I was watching a multi-level marketing campaign for Oracle development tools. I resolved to just teach myself how computers worked. That was the last computer science class I ever took.

Accelerated [Honors] Calculus lasts one year. By the end of the year, everyone with a shred of economic sense has quit to study Computer Science instead. The rest of us took Accelerated [Honors] *Advanced* Calculus in our second [sophomore] year of college.

My sophomore year of college was mostly "staying up after midnight in the library with my classmates doing calculus homework" and "working at the physics tutoring center where I helped other students with their calculus homework".

Was it worth it?

If college is about signaling to employers then "no". If college is about partying and finding a mate then "no, definitely not". If college is about learning things that make you smarter then "yes".

A physics student wrote an email to the tutors at my tutoring center. She had been getting bad scores on her exams and worried she wasn't smart enough for physics. I replied to her with an email that explained step-by-step what I did before each physics exam.

Her reply: "Oh. I guess I'm just not studying hard enough."
# Why is the war in Ukraine particularly dangerous for the world?

I have heard that since WW2 there have been ~140 armed conflicts in the world. Only few of them have been full-fledged wars, let alone big ones. However, as I don't know almost anything about wars and the politics around them, I cannot really formulate why this war is particularly dangerous and prone to start WW3. This one is specially frightening for me because it is very close by --I'm European--, but this does not make it more prone to e.g. start WW3. 

I have thought of few factors (location, countries involved / potentially involved, possession of nuclear weapons...) but the war in Ukraine does not seem to be too special in any of them. Is it maybe the combination of factors what it makes it particularly dangerous? As many may, like me, not be too versed in geopolitics/geostrategy, I think it would be useful to compile a list of key factors setting this war aside from all the other wars the world has endured since WW2 and making it particularly dangerous for the whole world.
# Patricia Hall & The Warlock Curse

Patricia Hall had studied all her life to become an archmage diviner of the Ivory Tower. When she was expelled, her heart broke and her world shattered. Even worse, she learned short after that the world might be literally about to end.

Join the eponymous character on a life-transforming journey that will take her out of academia to become an unlikely hero of magekind.

[**Read it now in our exclusive reading platform**](https://docs.google.com/document/d/1nLmKnOLLtGazArHItc6ejqnTMyZ2JlI2nzGQfIIrA5w/edit?usp=sharing)

* * *

### Testimonials

> Oh no, what have I done.

Jaime Sevilla, prolific writer

> I read the book! Really enjoyed it. Definitely very clear rationalist/EA influence.

Anonymous kind friend

> No Jaime, I haven't gotten around to reading your novella yet.

Anonymous patient friend

* * *

*Thanks to Faatima Osman, Anne Wissemann and Jonas Sandbrink for support and feedback.*

[**More from the author**](https://www.lesswrong.com/posts/cLBDeEnMsf4vfxvzc/counting-lightning)