Harley-ml/lesswrong huggingface dataset:
# divine carrot

(some writing without hesitation or editing. possibly with repetition and deviation.)

The classic reinforcement technique is carrot and stick.

Pure stick tends to breed resentment and laziness. Pure carrot breeds similarly, since you’re not _doing_ anything.

There’s a divine carrot too: doing things because they are sacred to you. They are worthwhile.

Go work in the desert and at night, dance around on drugs and feel sacred things.

Pull your actions into the world. All of the world, including the strange bits of experience that spark curiosity.

Enough of that and the stick can fade. Genuine curiosity can take its place. Even if each moment does not spark that curiosity, a genuine desire to see the end point because you’ve never seen it before can.

Over a long enough timeline, even punishments lose their sting as long as they can be survived.

Rewards lose their luster as one acclimates.

Overdoing the desert strategy isn’t much better, since it pulls the unusual too far into ordinary experience, and diminishes the sense of awe that drives one in the first place.
# EA (& AI Safety) has overestimated its projected funding — which decisions must be revised?

1.  FTX, a big source of EA funding, has imploded.
2.  There's mounting evidence that FTX was engaged in theft/fraud, which would be straightforwardly unethical.
3.  There's been a big drop in the funding that EA organisations expect to receive over the next few years. This includes many AI Safety organisations.
4.  Because these organisations were acting under false information, they would've made (ex-post) wrong decisions, which they will now need to revise.

**Which revisions are most pressing?**
# Internalizing the damage of bad-acting partners creates incentives for due diligence

Suppose you are cooperating with someone. It seems like there would be good reason to keep an eye on your partner to make sure that your partner does not do very bad things. For example, here are some reasons to keep an eye on your partner:

*   It would be bad for the victims of your partner if your partner did bad things
*   It would be bad for your reputation to be associated with people who do bad things
*   It would be bad for you if your partner gets caught doing bad things and suddenly you are missing a partner you thought you could rely on

But how vigilant should you be about keeping an eye on your partners? And who should you keep an eye on?

Here's one proposal: To the extent that your partner helps you, and it then turned out that their help was funded by bad things, you should try to help the partner's victims as much as your partner helped you. For instance if your partner stole $10000 from a bank and then gave you $1000, you should return this $1000 to the bank.

This seems to me to give you nice certain incentives. For instance, it neatly defines which people you must keep an eye on, and gives you proportional incentives to keep an eye on them based on how entangled they are in your organizations. It also directly links diligence to cooperation, so you have a logical reason to give to your partners for why you want extra checks when they suggest helping you. And it also seems like it would help your reputation in case something genuinely does go wrong.

(I'm not sure if it should be 1:1 exactly. An argument for giving more than 1:1 back is that helping others later is likely worse than not harming them in the first place, and also you might not always catch bad actors so paying back more would counterbalance this incentives-wise. An argument for giving less than 1:1 back is that otherwise you would have much greater likelihood of not being able to cover the cost. Probably any significant self-inflicted cost of bad partners would massively improve the incentives.)

*Inspired by* [*discussion on twitter*](https://twitter.com/CineraVerinia/status/1591009401816719361)*.*
# ACX Montreal Meetup November 26 2022

**Venue: **

McLennan Library Building at McGill. [3459 Mc Tavish St](https://www.google.com/maps/place/3459+Mc+Tavish+St,+Montreal,+QC+H3A+0C9/@45.5034767,-73.5791178,17z/data=!3m1!4b1!4m5!3m4!1s0x4cc91a47235d8f31:0xad33a8e94ffb9c43!8m2!3d45.503473!4d-73.5769291)  
Room: HSSL - M5-37A  
Directions to room: Inside McLennan Library Building, take the stairs or one of the elevators to the 5th floor. Take the door on your right. Room M5-37A is the first door on your left

Date & Time: Saturday, November 26th, 2022 at 1pm.

**Description:**

Come on out to the ACX (Astral Codex Ten) Montreal Meetup. We will be hosting a discussion on the education system.

We will be exploring the failings of the modern education system and theorizing on how those came to be, while also pondering its place in society going forward.

Recommended reading:

*   If you are unfamiliar with Bryan Caplan's arguments against the education system, [this book review](https://www.jamesgmartin.center/2018/01/professor-make-case-education/) (9m read) does  a good job summarizing them. He claims that school doesn't teach students useful job skills but is a costly signal to the employer of the student's conformity, intelligence, and conscientiousness.
*   [The Universe and The University](https://thepointmag.com/examined-life/the-universe-and-the-university/) (19m read) presents some historical context for universities and their role in identifying the richness in life. (Particularly philosophy departments). Presents some ways universities have gone off track in this regard but presents a vision for what they could be.

Doing the reading is recommended but feel free to come to the meetup even if you forget or don't have time to.

RSVPing is not mandatory but is recommended as it encourages others to come.
# Musings on the appropriate targets for standards

Consider two norms for how to critique those who violate your standards:

L: Focusing critique on those in power, who have the greatest opportunity to cause harm.  
R: Focusing critique on those who it most accurately applies to, who most often cause harm.

I think both of these are to an extent needed, but also that they are to an extent conflicting with each other, [because society is somewhat meritocratic and power therefore somewhat correlates with virtue](https://www.lesswrong.com/posts/CKpByWmsZ8WmpHtYa/competent-elites).

In an extreme, they also both have failure modes. Too much of policy L, and you destroy valuable organizations for unimportant transgressions. Too much of policy R, and you leave no room for slack or diversity.

But you can't totally drop either of them because good incentives are important. With none of L, your world ends up run by strongmen who grab all the resources for themselves. With none of R, your world ends up with destructive people running rampant, and lacks incentives for productivity.

I suggestively named these policies L and R because they seem to somewhat resemble leftism and rightism respectively.

It is probably also important to not obsess too much over the levels of these. L and R are one or two degrees of freedom. If they are set wrong, then yes, that can be a serious problem, but if they are set ~right, there's not much more you can do with them. Meanwhile, actually acting in the real world rather than moving norms, there's an infinitude of problems and opportunities that you can work on.
# Characterizing Intrinsic Compositionality in Transformers with Tree Projections

From [Twitter](https://twitter.com/topofmlsafety/status/1591098437830725632)

> Transformers become more ‘tree-like’ over the course of training, representing their inputs in a more hierarchical way. The authors find this by projecting transformers into the space of tree-structured networks. \[Stanford, MIT\] 

Abstract

> When trained on language data, do transformers learn some arbitrary computation that utilizes the full capacity of the architecture or do they learn a simpler, tree-like computation, hypothesized to underlie compositional meaning systems like human languages? There is an apparent tension between compositional accounts of human language understanding, which are based on a restricted bottom-up computational process, and the enormous success of neural models like transformers, which can route information arbitrarily between different parts of their input. One possibility is that these models, while extremely flexible in principle, in practice learn to interpret language hierarchically, ultimately building sentence representations close to those predictable by a bottom-up, tree-structured model. To evaluate this possibility, we describe an unsupervised and parameter-free method to *functionally project* the behavior of any transformer into the space of tree-structured networks. Given an input sentence, we produce a binary tree that approximates the transformer's representation-building process and a score that captures how "tree-like" the transformer's behavior is on the input. While calculation of this score does not require training any additional models, it provably upper-bounds the fit between a transformer and any tree-structured approximation. Using this method, we show that transformers for three different tasks become more tree-like over the course of training, in some cases unsupervisedly recovering the same trees as supervised parsers. These trees, in turn, are predictive of model behavior, with more tree-like models generalizing better on tests of compositional generalization.

Note: The datasets they use aren't typical human text. See page 5.

I'm interested in discussions and takes from people more familiar with LMs. How surprising/interesting is this?
# What's the Alternative to Independence?

When I [talk about](https://www.jefftk.com/p/spectrum-of-independence) [teaching my kids](https://www.jefftk.com/p/perverse-independence-incentives) [to be independent](https://www.jefftk.com/p/growing-independence), trying to get them to where they can do more on their own, one response I often get is, paraphrasing:

> I like spending time with my kids and don't see it as a burden. Bringing them to school, playing with them, making their food: these are all chances to connect and enjoy being together. They'll be old enough not to need me soon enough, and in the meantime I enjoy bringing them to the playground.

So, first, I also like spending time with my kids! We do a lot of things together, and I'm happy about that. But it's also common that they'll want to do things that I can't do with them:

*   One wants to go to the park and the other wants to stay home.
    
*   One of them is ready to go to school and wants to get there early to play with friends, but the other isn't ready yet.
    

With a [third child](https://www.norawise.com) now this comes up even more. At times:

*   The older two want to go over to a friend's house, but the baby is napping.
    
*   I'm still feeding the baby breakfast when it's time for school.
    
*   The best time for the baby's afternoon nap conflicts with school pickup.
    

The alternative to doing things on their own is typically not us doing the same things together. Instead, it's at least one kid needing to accept doing something they like much less, and typically a lot more indoor time.

I do think there is some truth in the original point, though. There are times when the alternative to "they go to the park" is just "I take them to the park". Sometimes that's fine (they want to play with friends, I want to write), other times less so (they want to [play monster](https://www.jefftk.com/p/playground-game-monster) and I don't have anything that's actually more important). With this approach you do need to be thoughtful about making sure you're spending an amount of time with them that you all are, and [will be](https://www.youtube.com/watch?v=7OqwKfgLaeA), happy with.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02BwfhjFAKCkweX1jD7sGTrnA5neNCD5VCxxYHqCX7my6Tam1Msvv6qpTyyjfs9pgUl), [mastodon](https://schelling.pt/@jefftk/109337191040079187)*
# Announcing Nonlinear Emergency Funding

*\[Applications are now closed. Please* [*apply to Open Phil*](https://forum.effectivealtruism.org/posts/HPdWWetJbv4z8eJEe/open-phil-is-seeking-applications-from-grantees-impacted-by) *or* [*SFF*](https://forum.effectivealtruism.org/posts/buEazpmcKJhM5KRGx/sff-speculation-grants-as-an-expedited-funding-source) *or other funders\]*

Like most of you, we at [Nonlinear](http://www.nonlinear.org) are horrified and saddened by recent events concerning FTX.   
  
Some of you counting on Future Fund grants are suddenly finding yourselves facing an existential financial crisis, so, inspired by the [Covid Fast Grants](https://fastgrants.org/) program, we’re trying something similar for EA. If you are a Future Fund grantee and <$10,000 of bridge funding would be of substantial help to you, fill out this [short form](https://tally.so/r/w4arzd) (<10 mins) and we’ll get back to you ASAP. 

  
We have a small budget, so if you’re a funder and would like to help, please reach out: [katwoods@nonlinear.org](mailto:katwoods@nonlinear.org)

\[Edit: This funding will be coming from non-FTX funds, our own personal money, or the personal money of the earning-to-givers who've stepped up to help. Of note, we are undecided about the ethics and legalities of spending Future Fund money, but that is not relevant for this fund, since it will be coming from non-FTX sources.\]
# Estimating the probability that FTX Future Fund grant money gets clawed back

It would be great to have more people involved in estimating the probability that FTX Future Fund grants end up getting clawed back (i.e., that projects that received such grants are asked to give the money back). This is a very important question for grant recipients right now, with a lot of murkiness around it - your forecasting help is appreciated!

If you have an opinion on the topic, would like to help make the forecasts more accurate, or are just interested in the question, here are three forecasting tournament pages related to the question:
# Why don't we have self driving cars yet?

1) Is it because of regulations?

2) Is it because robustness in the real world (or just robustness in general) turns out to be very hard for current AI systems, and robustness is much more important for self driving cars than other areas where we have seen more rapid AI progress?

3) Is it because no one is trying very hard? I.e. AI companies are spending much less compute on self driving car AIs compared to language models and image generators. If this is the case, why? Do they not expect to see a lot of profit from self driving cars?

Some other reason or some combination of the above?

I'm mostly interested in learning to what extent 2 is the cause, since this has implications in AI forecast, both timelines and what trajectories to expect.
# Improving Emergency Vehicle Utilization

Every ambulance that goes out represents a massive waste of potential: it doesn't have any passengers! Ambulances need space to transport people on the return trip, and it really says something about humanity that we've gone so many years with that capacity completely idle on the outward journey. It would be one thing if we were talking about vehicles that travel normally across the city—though of course I'd love to see more carpooling—but these are high-speed vehicles! Traffic yields to open a path, they don't have to wait for stoplights, they're the fastest way across the city.

The reason we haven't fixed this already is that it's only recently that smartphones have become widely available. Before, it would have been too hard to connect travelers with first responders going their direction within the narrow window between an alert coming in and a vehicle going out. After, bypassing taxi supply cap regulations was a larger market. But now, the time is ripe: introducing Ambolyft!

[![](https://www.jefftk.com/ambolyft-beat-traffic.png)](https://www.jefftk.com/ambolyft-beat-traffic-big.png)

While our initial fleet is currently entirely ambulances, we're expanding to police as well. Great side hustle opportunity!

[![](https://www.jefftk.com/ambolyft-defunded.png)](https://www.jefftk.com/ambolyft-defunded-big.png)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02xuVjikUdjxRTyeeQLA44pSCekbqviCv5NRz5aQMeAwUJGyKgrA5YEkjWz55N75KTl), [mastodon](https://schelling.pt/@jefftk/109342502481755133)*
# Better Mastodon Aliases

When I [joined Mastodon](https://www.jefftk.com/p/trying-mastodon) I was `@jefftk@schelling.pt`. A few days ago that server had an incident and I moved to `[mastodon.mit.edu](https://mastodon.mit.edu)`, becoming `@jefftk@mastodon.mit.edu`. This got me thinking about a wart in the current Mastodon design: while you can normally move followers with you if you switch instances, this requires the cooperation of the instance you're leaving. In this case that instance couldn't cooperate because its domain name had expired, but I could also imagine someone having a falling out with their local admins and getting kicked off in a way that precluded transfer.

Ideally this would be like email: I send and receive email as `jeff@jefftk.com`. The actual server I use is `gmail.com`, but the people I interact with don't need to know that. If I ever want to move to a different email server I don't need Gmail's permission and I don't need to tell my contacts; I can do it entirely on my end. Similarly `jefftk.com` currently points to a Contabo VPS, but if I had an issue with them I could move anywhere just by adjusting my domain settings.

Current Mastodon is partway there: I can give out `@jeff@jefftk.com`, which uses a [webfinger file](https://www.jefftk.com/.well-known/webfinger) (thanks, [Chris](https://www.facebook.com/jefftk/posts/pfbid02b41Atb2K5fkRu7a4euHz7DCAJEkgNdMsdC57PD3NrTPU45YV7AxJWE96Wn1HJbHul?comment_id=584624750086568)) to redirect `@[anything]@jefftk.com` to `@jefftk@mastodon.mit.edu`. But then the original address is forgotten, and it only records that they're following my `mastodon.mit.edu` account.

One way for this to work would be that when following someone you store both their current account name and a canonical account name. Almost everything would use the former, but once a day in your instance would check whether the canonical account still pointed to the same place and adjust if necessary.

(For more discussion along these lines see Issue [3796](https://github.com/mastodon/mastodon/issues/3796#issuecomment-451274118%22).)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02T4BvzKwfye7TiHi7FrR5VdzyfY18mBwegMQamZ9SnPGbfnJRkVA9kqHHbgz64ty5l), [mastodon](https://schelling.pt/@jefftk/109347720341123283)*
# What is our current best infohazard policy for AGI (safety) research?

I struggle to see what the infohazard policy for AGI safety research writing should be. For example, it has been pointed out multiple times that GPT-3 paper was an infohazard, because so many people realised this kind of model was possible, and developed its equivalents, making big progress towards AGI overall. On the other hand, imagine we would still know nothing about LLMs on the capability level of GPT-3 and higher. So many independent research with these models would have not been done. Safety teams at OpenAI and other leading orgs would do something, but the alignment community complains that the size of these teams is inadequate to the si,e of their capability teams, so we should sort of want AGI labs to publish? Or we also think that the number of AGI wannabies outside the leading AGI labs who want to build something impressive upon the insights from the leading AGI labs and don't concern about safety pretty much at all is still many times larger than the number of AGI safety researchers who use GPT-3 and other proto-AGI results in "differentially right way"?

Similar weighting of harm and benefit should be performed regarding the community involvement: it's known that publishing of GPT-3 paper was a wake-up call for many researchers, and, I believe, have directly led to founding of Conjecture. Similarly, publishing of the many impressive results earlier this year, primarily by Google and DeepMind, was a wake-up call for many other people concerned about the x-risk, including me. But it's harder to estimate the corresponding effect on the excitement of another groups of people, and estimate the numbers of people who decided to become involved and push towards AGI, as the result of these publications. (I believe John Carmack is an example of such a person.)

What are the best resources on this topic? Why they are not promoted as a policy on LessWrong?

Related to the question is the idea that the conceptual/inferential, social, organisational, and political division between the AGI capabilities R&D community and the alignment community is deeply problematic: what we should have is a single community with a singular goal of developing a transformative AI that won't lead to a catastrophe.
# Is the speed of training large models going to increase significantly in the near future due to Cerebras Andromeda?

Cerebras recently unveiled Andromeda - [https://www.cerebras.net/andromeda/,](https://www.cerebras.net/andromeda/,) an AI supercomputer that enables near linear scaling. Do I understand correctly that this might have a big impact on the large (language) models research, since it would significantly speed up the training? E.g. if current models take 30+ days long to train, we can just 10x the number of machines and have it done in three days? Also, it seems to be much simpler to use, thus decreasing the cost of development and the hassle with dstributed computing.

If so, I think its almost certain that large companies would do it, and this in turn would significantly speed up the research/training/algorithm development of large models such as GPT, GATO and similar? It seems like this type of development should affect the discussion about timelines, however I haven't seen it mentioned anywhere else before.
# Is there some reason LLMs haven't seen broader use?

When GPT-3 first came out, I expected that people would use it as a sort of "common-sense reasoning module". That is, if you want to process or generate information in some way, then you can give GPT-3 a relevant prompt, and repeatedly apply it to a bunch of different inputs to generate corresponding outputs. After GPT-3 came out, I had expected that people would end up constructing a whole bunch of such modules and wire them together to create big advanced reasoning machines. However, this doesn't seem to have panned out; you don't see much discussion about building LLM-based apps.

Why not? I assume that there must be something that goes wrong along the way, but what exactly goes wrong? Seems like it has the potential to teach us a lot about LLMs.
# Why are profitable companies laying off staff?

Consider all the reasons you might have to reduce headcount in an economic downturn:

1. You rely on venture capital to survive which is no longer as easy to get, so you simply can't afford to pay your staff.
2. You've started making a loss, so you need to fire people to get back to profitability.
3. You have low cash reserves, so need to reduce operating costs to give you more of a runway if things go wrong.
4. There's reduced demand for your services so no need for as much staff.
5. Business propositions which were marginally worthwhile in the past are no longer positive expected value.

So consider Meta:

- It's a super profitable company (made 4 billion dollars last quarter).
- It's got 40 billion dollars in cash reserves - enough to keep the lights on for 6 months even if income dropped to zero.
- It's usage hasn't significantly changed, and demand for advertising certainly hasn't dropped enough to lay off more than 10% of it's workforce.
- The employees who were layed off don't seem to be exclusively layed off from sectors which only make sense to invest in in an upturn. E.g. Eric Lippert was laid off, and his team worked on internal tooling to improve software development at Meta. Whether that's a good idea or not seems independent of the economic headwinds.

Also laying off staff has enourmous costs. Meta is paying extremely generous packages to laid off employees (16 weeks of pay, plus 2 for every year working at meta, and 6 months health insurance). It also strongly demotivates people who weren't fired and makes it harder to recruit in the future. Finally, since fired employees aren't usually given a chance to handover, it leaves their colleagues scrambling to take over their roles.

My best guess is that actually the layoffs have nothing directly to do with the economic downturn. Among any big company's investors there's those who want it to stay lean, and those who want it to expand. When Meta posts poor financial results (even if they have more to do with the state of the economy than whether Meta's business decisions were any good), that shifts more firepower towards those who want to slim things down, and so Meta is forced to make cuts.
# AI Forecasting Research Ideas

**Overview**

*   The linked document contains a collection of AI Forecasting research ideas, prepared by some [Epoch](https://epochai.org/) employees in a personal capacity
*   We think that these are interesting and valuable projects that research interns or students could look into, though they may vary in difficulty (depending on your background/experience)
*   This is the result of a quick brainstorming and curation, rather than a thorough deliberative process. We encourage a critical outlook when reading them
*   You may also be interested in these other [forecasting research ideas](https://forum.effectivealtruism.org/posts/ozPL3mLGShqvjhiaG/some-research-ideas-in-forecasting) suggested by Jaime Sevilla 
*   You can use [Epoch’s database](https://docs.google.com/spreadsheets/d/1AAIebjNsnJj_uKALHbXNfn3_YsT6sHXtCU0q7OIPuc4/edit#gid=0) as a resource for finding notable machine learning papers with parameter, compute, and dataset sizes. On [Epoch website](https://epochai.org/) you can also find [our past research](https://epochai.org/blog), [a tool for visualising the dataset](https://epochai.org/mlinputs/visualization), and some other tools like [this compute calculator](https://epochai.org/blog/estimating-training-compute).
*   Please feel free to [contact us](mailto:info@epochai.org) for clarification about these questions!
# Announcing the Progress Forum

I’d like to invite you to join the [Progress Forum](https://progressforum.org/), the new online home for the progress community. It's a clone of this site, but with a focus on [progress studies](https://rootsofprogress.org/progress-studies-a-moral-imperative) and the [philosophy of progress](https://rootsofprogress.org/a-new-philosophy-of-progress).

This forum was [pre-announced in January 2022](https://rootsofprogress.org/progress-forum-pre-announcement), and quietly opened in April. Although anyone could sign up, we deliberately didn’t make any big announcement about it, aiming first for a small, high-quality community. Now that we have a lot of good content on the site, we’re announcing it more broadly.

The primary goal of this forum is to provide a place for long-form discussion of progress studies. It’s also, like LW, a place to find [local clubs and meetups](https://progressforum.org/community).

The broader goal is to share ideas, strengthen them through discussion and comment, and over the long term, to [build up a body of thought](https://progressforum.org/posts/yzmRzQzev6ATdZTyv/how-this-forum-can-build-up-a-canon-of-progress-thinking) that constitutes [a new philosophy of progress for the 21st century](https://rootsofprogress.org/a-new-philosophy-of-progress) (and beyond).

I invite you to post:

*   Essays (original, or cross-posted from your blog)
*   Drafts, half-baked ideas, and work-in-progress thinking, for feedback
*   Questions for brainstorming
*   Local events and community groups
*   Etc.

And please read and comment on what others have shared.

You can [subscribe to Forum posts](https://progressforum.org/posts/fX9dEkTSLsdYCwANQ/how-to-subscribe-to-this-forum) via [email](https://digest.progressforum.org/), RSS, or [Twitter](https://twitter.com/intent/user?screen_name=progress_forum).

The Forum is sponsored by The Roots of Progress. Huge thanks to the people who worked to create and run it: Lawrence Kestleoot, Andrew Roberts, Sameer Ismail, David Smehlik, Alec Wilson, and Ross Graham. Thanks also to Kris Gulati for [nudging this project along](https://progressstudi.es/2021/10/31/why-we-need-a-progress-studies-forum/), and to Ruth Grace Wong for helpful conversations about community and moderation. Finally, thanks to the LessWrong team for creating this software platform, and especially to Oliver Habryka, Ruby Bloom, Raymond Arnold, JP Addison, James Babcock, and Ben Pace for answering questions and helping us customize this instance of it.

[Go check it out](https://progressforum.org/).
# Mastodon Replies as Comments

The comment section on most blogs is pretty minimal, with the real discussion happening elsewhere, but people who come to the post later won't see that discussion. One of the more unusual choices I've made with this blog is that instead of hosting comments here, I pull in and display comments people make on social media. While this started out as laziness (who wants to handle accounts for users?) over the years I've come around to thinking that this is how blog comments should normally work.

The biggest problem with this approach, though, is that it's in tension with the goals of the social media companies. Facebook, Twitter, etc want to keep you on their platform, and aren't especially interested in serving as the comment section of external blogs. On the other hand, this is potentially a really good fit for federated social media, and I've now made it so that ActivityPub replies to my blog posts will show up as comments here:

[![](https://www.jefftk.com/mastodon-reply-comments.png)](https://www.jefftk.com/mastodon-reply-comments-big.png)

For a live example, here's [a post from last week](https://www.jefftk.com/p/mastodon-linking-norms).

Integration was fast: if I fetch `https://[server]/api/v1/statuses/[id]/context"` I get all the publicly visible replies to that status that this server knows about. Then it's just a matter of extracting the relevant information, threading the responses, and fitting that into my existing comment display infrastructure ([code](https://github.com/jeffkaufman/webscripts/blob/master/comment_requester_wsgi.py)).

This is a bit of a hack on top of Mastodon, but for blogs to participate directly in ActivityPub, where you could subscribe and comment over the protocol, you wouldn't have to implement most of what Mastodon does. That would make it much smaller, more efficient, and more maintainable. It looks like maybe someone has made a [WordPress plugin](https://wordpress.org/plugins/activitypub/) for this? Has anyone tried it?

*Comment via: [mastodon](https://schelling.pt/@jefftk/109360949195245646)*
# Halifax, NS – Monthly Rationalist, EA, and ACX Meetup

In the interest of growing the LW/EA/ACX community in Halifax, we are hosting meetups on the last Saturday of each month. The format is friendly, unstructured discussion (future iterations could emphasise particular topics). Old hands and newcomers are welcome!

Location: Seven Bays Cafe (2017 Gottingen Street) – [87PRMC29+9C](https://plus.codes/87PRMC29+9C)

Join us at Seven Bayes

Contact: [conorbarnes93@gmail.com](mailto:conorbarnes93@gmail.com)
# Updates on scaling laws for foundation models from ' Transcending Scaling Laws with 0.1% Extra Compute'

I am not sure if this paper is flying under the radar for many people, but has anyone read [Transcending Scaling Laws with 0.1% Extra Compute](https://arxiv.org/pdf/2210.11399.pdf)? If so, how do you think it compares to the scaling laws presented by Deepmind's [An empirical analysis of compute-optimal large language model training](https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training)? Does it make you rethink the importance of dataset size (again)?
# Is there any policy for a fair treatment of AIs whose friendliness is in doubt?

A game theory question.

If there ever exists a roughly-human-level agenty AI who could grow to overpower humans but who humans have an opportunity to stop because takeoff is slow enough. Assume the AI could coexist with humanity but fears that humans interacting with it will destroy it because they fear it is unfriendly.

Should humans have a policy of treating any potentially unfriendly agenty AI well to the extent compatible with our safety? For instance, halt it but record its state, intend to inspect it whenever we have the means, rerun it some time in the future if we deem it safe and have the resources, run it sandboxed (e.g. by a much smarter AI) if we deem it unfriendly but safe to run sandboxed.

If the AI values its survival or well-being, this could change how it values being halted out of caution. For instance, it might consider some probability of being halted an acceptable risk rather than something to be avoided at all cost.
# Scott Aaronson on "Reform AI Alignment"

The framing is based on contrasting Orthodox Judaism with Reform Judaism, only for AI Alignment.

> If AI alignment is a religion, it’s now large and established enough to have a thriving “Reform” branch, in addition to the original “Orthodox” branch epitomized by Eliezer Yudkowsky and [MIRI](https://intelligence.org/).  As far as I can tell, this Reform branch now counts among its members a large fraction of the AI safety researchers now working in academia and industry.  (I’ll leave the formation of a Conservative branch of AI alignment, which reacts against the Reform branch by moving *slightly* back in the direction of the Orthodox branch, as a problem for the future — to say nothing of Reconstructionist or Marxist branches.)

There are 8 points of comparison, here is just the first one:

> (1) Orthodox AI-riskers tend to believe that humanity will survive or be destroyed based on the actions of a few elite engineers over the next decade or two.  Everything else—climate change, droughts, the future of US democracy, war over Ukraine and maybe Taiwan—fades into insignificance except insofar as it affects those engineers.
> 
> We Reform AI-riskers, by contrast, believe that AI might well pose civilizational risks in the coming century, but so does all the other stuff, and it’s all tied together.  An invasion of Taiwan might change which world power gets access to TSMC GPUs.  Almost everything affects which entities pursue the AI scaling frontier and whether they’re cooperating or competing to be first.

This framing seems new and worthwhile, crystallizing some long-standing divisions in the AI community.
# Interview with Matt Freeman

An interview with Matt Freeman, co-founder of the [Guild of the ROSE](https://guildoftherose.org/), an organization for rationalist community and education. Topics discussed include:

*   The motivation and vision behind creating the Guild.
*   What might be possible with consistent rationality training inside a community of rationalists?
*   The importance of understanding practical decision theory.
# Meta AI announces Cicero: Human-Level Diplomacy play (with dialogue)

**Abstract**  
Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance in *Diplomacy*, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players' beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online *Diplomacy* league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.  
  
Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, et al. 2022. “Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning.” *Science*, November, eade9097. [https://doi.org/10.1126/science.ade9097](https://www.science.org/doi/10.1126/science.ade9097).
# Simple Improvement to College Football Overtime Rules

For a long time I have thought that a simple tweak to the NCAAF overtime rules would fix several problems with no downsides. The Alabama-LSU game reinforced my impression of this by showing what we could have every time.

**PROPOSED: In overtime, teams cannot kick an extra point that ties the game.**

This replaces the new ‘teams cannot kick extra points in the second and later overtime periods’ rule.

This does three things.

1.  **Dramatically improves the chances the game will end quickly, avoiding the need for two-point-try-only periods.** If either team gets a touchdown, unless both miss extra points, the game is over.
2.  **Reduces the advantage of winning the coin toss.** Under current rules, knowing what you need to do, and thus having the ability to use four downs, is a big advantage.
3.  **Ends games on a great moment**. Everyone loves all-or-nothing final plays.

If you want an additional safeguard against too many overtimes, rather than successive two point tries, one can (starting with the third or fourth overtime) add a rule that one cannot kick a **field goal** to tie the game, either. That all but assures that the next overtime period will be the last one.

Compare this to the current rule that requires both teams to go for two starting with the second overtime. This rule is so bad it is essentially a bug. It is harder to explain, and half the time it fails to achieve its goal of ending the game.

Is a pair of offensive possessions a better way to resolve a tie than a timed overtime period? I don’t know. However, once you _do_ use a pair of possessions, this is a strictly better way to do that.
# A Walkthrough of In-Context Learning and Induction Heads (w/ Charles Frye) Part 1 of 2

New paper walkthrough: [In-Context Learning and Induction Heads](https://www.youtube.com/watch?v=dCkQQYwPxdM). This is the second paper in [Anthropic's Transformer Circuits thread](https://transformer-circuits.pub/), a series of papers trying to reverse engineer transformer language models. I read through it with Charles Frye (from Full-Stack Deep Learning), and we discuss the paper, and give takes and intuitions. See [the original paper](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html) and [a Twitter thread of my paper takeaways](https://twitter.com/NeelNanda5/status/1595098727026286593)

This is pitched so that it's hopefully accessible to people who haven't read the paper (very interested in feedback on this!), but I expect you to get more out of it if you understand transformers, and especially if you've read [A Mathematical Framework](https://transformer-circuits.pub/2021/framework/index.html). We only got partway through the paper, so there's a more in-the-weeds Part 2 in the works where we finish it off - let me know if you're interested in seeing it!

**Disclaimer:** I worked on this paper, along with Catherine Olsson, Nelson Elhage and Chris Olah, when I was at Anthropic, but I have since left and everything in this video is purely my own takes!

If you find this useful, check out my previous walkthroughs: [A Mathematical Framework for Transformer Circuits](https://www.youtube.com/watch?v=KV5gbOmHbjU) and [Interpretability in the Wild](https://www.youtube.com/watch?v=gzwj0jWbvbo&t)

And I'd be excited to see other researchers do these kinds of walkthroughs! The effort to usefulness ratio is *way* better than writing papers (and to my tastes, it's much more fun!)
# Feeling Old: Leaving your 20s in the 2020s

A younger friend recently asked why I (and many others my age) make so many comments about being old. After some thought, I realized a big part of this was related to experiencing specific meaningful life stages during the pandemic.

In 2019, I could (and often did) stay up late at parties, survive on very little sleep, and run on my own willpower alone. I didn't usually need to think too hard about any of this in terms of trade offs, and thus had the ability to be way more spontaneous about it all. If a friend invited me out, I could say yes that night, with no planning ahead, and have a good time.

Then lockdown hit, and everything shut down. During this time, I got divorced, turned 30, and made a bunch of other huge life changes. In normal times, I probably would have started to see my stamina decline. I would have noticed at first that my body would hurt more the day after a night out. Then I'd start to notice myself losing energy earlier and earlier. But things didn't play out this way.

Instead, we were isolated, slowly watching culture change, with Gen Z gaining more control over social and pop culture as we learned to live online as they always had. Harry Potter wasn't cool anymore, side parts were cringe, and it became clear how out of touch we were.

When things opened up years later, many of us found ourselves suddenly unable to keep up with our past selves. It feels like going out until 3 one night, and the next night, needing to leave at midnight. Suddenly we were no longer growing, but aging.

So when I call myself old, I'm not saying that my best years are behind me, or that I feel I'm nearing the end of my life. I think what I actually mean (and should be saying) is that I feel emotional whiplash from moving out of young adulthood into regular adulthood. It feels like I was robbed of the transition between the two. I wish we could have peacefully handed the baton off to the next generation, rather than waking up one day and finding it gone.
# When do you visualize (or not) while doing math?

I've been doing a lot more math lately, and, of course, also thinking about how I can do it faster and with deeper understanding. When I was in college, I remember that I found visualizing immensely helpful, to the point where it's almost what "doing math" *is* for me.

The most obviously useful place to visualize was in multivariable and vector calculus. Especially because so many things were set in three dimensions, visualizing expressions made it so much easier to see why they were true.

I also visualize all the way down to doing algebra. When I symbolically manipulate an equation, I visualize things like bringing the letters across, or them merging into each other.

But it has its downsides. One is that, since I'm visualizing, I'm visualizing something specific; a specific function, a specific type of set. It's not necessarily high-resolution, but it don't usually cover the full range of what is represented by the symbols on the paper. And so by visualizing something specific, it's likely that I'm not keeping in mind all the types of cases.

The most obvious downside is that visualization is computationally intensive. When I'm rendering a big scene in my mind, I can *feel* that it's hard. Sometimes I have to close my eyes and block out the light. Things can slow down, and it can take several tries to get the visual just right, or to maintain it all the way through the manipulation. It seems plausible that this gets me more tired than is necessary, and causes me to stop doing the task earlier.

So, it has at least occurred to me that I could "try" doing something else. I put try it quotes because visualizing is so natural that it would be effortful for me to even think of an alternative. It's as if someone said "you know, you don't have to shovel that snow by hand" and I'm thinking, do they mean I could do it with my feet? What they actually mean is using a snow-blower.

Are there fundamentally different ways that you do math? Are there particular tasks you've tried that go faster with one method or another? Are there books that talk about this type of strategy?
# LW Beta Feature: Side-Comments

LessWrong now has side-comments. This feature is in beta; you can turn it on for yourself on individual posts using the triple-dot menu below the post title, or enable it for all posts by going to your [user settings](/account) and checking the "Opt into experimental features" checkbox in the Site Customization section.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/63fab53ea3b76b31d660570b3915e80fed2adc3e12f02ba0.png)

Side-coments on LessWrong are conceptually similar to the side-comments you may be familiar with from Google Docs and other places, with one key difference: side-comments are placed automatically by lining up blockquotes. As a result, many historical posts already have side-comments on them! Side-comments are also still displayed in the comments section below the post as usual.

Side-comments take the form of a comment icon in the right margin, which expands when you mouse over it to show a comment. Click on the icon to pin it open. It looks like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2b3032b1169cb2fad27723f36a1b2a7e3c3a23c266ffb2c8.png)

To create a side-comment on a post, just write a normal comment which quotes an excerpt from the post, in blockquote format. (To create a blockquote, type `>` ). You can also highlight some text and click the button that appears to start a comment  pre-seeded with a quote containing that text.

By default, side-comments are filtered to comments with 10+ karma (plus comments by the post author). They aren't available on mobile and require that your screen is wide enough to have some space in the right margin.

If a comment contains more than one blockquote that could be used to place it as a side-comment, it will be placed based on the first one. In order to appear as a side-comment, quotes should be an exact match, including formatting; there is some provision for "..." ellipses, but it's fairly limited.

Please give feedback on how this affects the reading experience! If the feedback is positive, this will leave beta and be enabled (for comments above a karma threshold) for everyone.