Harley-ml/lesswrong huggingface dataset:
# Open application to become an AI safety project mentor

EffiSciences is a French organization committed to a more involved research to address the pressing problems of this century.

We are organizing bootcamps for French university students in AI safety. After teaching an intensive curriculum (PyTorch, ML, interpretability, transformers, conceptual AI safety), students start a research project. We also organize research group et seminars that end on research projects in top French universities.

We are looking for qualified mentors to supervise groups of 1-3 students on a technical or conceptual AI safety project. Supervision consists of a 1 or 2 hours session per week for about 3 months.

Our students come from the best French engineering schools and universities and will be familiar with the concept of the Alignment problem. We think that your contribution with this supervision would have a significant chance to create a new AI Safety researcher, as well as give you some experience in mentoring bright students.

If you have any doubts about your availability or skills, we still encourage you to apply using the following form. We will most likely contact you in any case.

[https://docs.google.com/forms/d/e/1FAIpQLScghGTf_mMHzllSgXhbWQD91wgY2ckDVnJSbNGMHeuhcBuFFw/viewform](https://docs.google.com/forms/d/e/1FAIpQLScghGTf_mMHzllSgXhbWQD91wgY2ckDVnJSbNGMHeuhcBuFFw/viewform)
# [Link] "Improper Nouns" by siderea

The post:

- [siderea | Improper Nouns [p/a/s, Patreon]](https://siderea.dreamwidth.org/1773806.html)

Interestingly, 'improper nouns' – as described by the post – is itself an example of the phenomena it describes (as 'improper noun' is itself an existing term):

> There's a thing you probably have encountered and maybe haven't really registered as it went by. It usually plays out like this. Somebody else takes a noun phrase – usually there's an adjective or two involved, sometimes multiple nouns – that seems perfectly ordinary and makes sense on the face of it, and which refers to something in the real world, and the speaker makes some bold assertion about it that strikes you as *really weird*, and possibly highly prejudicial. What they've said leaves you thinking, "wait, how could that *possibly* be true?"
> 
> That's all you get. That's all the sign of what's happening. That's the glitch in the matrix.
> 
> There's a lot of reasons that people make ridiculous claims about all sorts of things, and you will always be tempted to write it off as just ridiculous people saying ridiculous things and thinking no more about it.
> 
> But if you can catch yourself in that moment, and *not* reflexively dismiss it as silliness, you could, instead, ask yourself: "Hey, is that noun phrase maybe a **technical term of art** which means something much more specific and possibly quite different from what the individual words literally mean?"
> 
> Doing this? Is a *super power*. Lots of people can recognize specific technical terms that they are acquainted with. Few people cultivate the more general ability to recognize – or just suspect – that a noun phrase is a technical term they *aren't* familiar with.

This is a useful enough idea that it's already been referenced in [another post](https://www.lesswrong.com/posts/L9CeB4huNtYzT4QrT/interpreting-affordable-housing) here.
# Resources to find/register the rationalists that specialize in a given topic?

As far as I can tell:

1.  It's not possible for me (or most/all other people, for that matter) to individually evaluate claims every topic I hear about, as it takes too long time and too much effort to learn the foundations needed to evaluate things accurately. Therefore I (and all other people?) need to rely on other people to evaluate topics that I am unfamiliar with.
2.  On a lot of topics, common experts are incompetent; for instance most social science research is just bad, and it's mainly bad because the entire fields have no clue what they're doing. Therefore it seems unwise to rely on those fields. It's unclear why they are so bad, but IME rationalist experts are much better, perhaps because they share my biases, perhaps because rationalism teaches useful tricks, but I think more likely because rationalism just tends to attract incredibly intelligent and curious people.
3.  However, one cannot just blindly rely on opinions by rationalists, as rationalists are not automatically good at everything; when rationalists are unfamiliar with a topic, they often say lots of totally clueless stuff.[^ry6fox4z9ei]

So in my view there's basically a lot of utility in finding the rationalists most specialized in each topic to make it easy to learn stuff from them. Has anyone worked on this in the past? Is this something someone (Lightcone Infrastructure, perhaps?) would be interested in setting up?

And I guess if you know of any underrated rationalists who specialize in some topic, feel encourage to share in the comments for this post?

[^ry6fox4z9ei]: This also seems to apply to non-rationalists, but that's not as important for this purpose.
# What's the actual evidence that AI marketing tools are changing preferences in a way that makes them easier to predict?

I've encountered this claim multiple times over the years (most recently on [this AXRP episode](https://axrp.net/episode/2021/02/17/episode-4-risks-from-learned-optimization-evan-hubinger.html)), but I can't trace its origins (it doesn't seem to be on Wikipedia). Quoting Evan from the episode:

> And so if you think about, for example, an online learning setup, maybe you’re imagining something like a recommendation system. So it’s trying to recommend you YouTube videos or something. One of the things that can happen in this sort of a setup is that, well, it can try to change the distribution to make its task easier in the future. You know, if it tries to give you videos which will change your views in a particular way such that it’s easier to satisfy your views in the future, that’s a sort of non-myopia that could be incentivized just by the fact that you’re doing this online learning over many steps.  
>   
> And if you think about something, especially what can happen in this sort of situation is, let’s say I have a - Or another situation this can happen is let’s say I’m just trying to train the model to satisfy humans’ preferences or whatever. It can try to modify the humans’ preferences to be easier to satisfy.

Furthermore, there's a world a difference between deliberately optimising for modifying preferences in order to make them easier to predict, vs preferences changing as a byproduct of the AI getting better at predicting them and thus converging on what to advertise. This matters for what predicted features of strategies an AI is likely to pick out of strategy space when new options are introduced.
# What sorts of preparations ought I do in case of further escalation in Ukraine?

I guess the Russian invasion of Ukraine can go in many ways. It might just fizzle out, or continue to be a fairly conventional war with extent limited to Ukraine, or it might escalate with a Russian nuclear strike against Ukraine followed by NATO counterattacks against Russia, and it doesn't seem implausible to me that it could spin totally out of control into a WW3 scenario. Or various other paths.

The more extreme end of the range with these scenarios seem worth preparing for, but I'm not completely sure what preparations would be the most relevant. I thought LessWrong might have some good ideas.

If relevant, I live in Denmark, a NATO country close to the core of the EU, not quite at the border of Russia but also not quite buffered away from Russia (especially if counting Kaliningrad). Though preparation ideas that are more relevant to people in other countries than Denmark are welcome too as they might be useful to someone else.
# Company leadership and tactical decisions

The other day at work, I received an email from our leadership regarding "new principles for our organization", linking out to a web page listing the new principles.  This ended up being partially a culture document, but mostly a direction document - "we want customers to be able to X" "we want to provide industry standard Y", etc.  While the wish list was pretty explicit and fairly detailed, there were no concrete actions.

At the level that our leadership works at, I believe this is largely the correct thing to do.  If you're trying to set direction for a thousand people, you can't really specify technical details, and even trying to promote specific strategies is almost always going to end up with suboptimal results. Plus, if you hire smart, motivated people, they'll find good (or good enough) solutions.

Except in the case of a systemic defect or dysfunction which keeps most or all of your teams trapped in local minima, while there's a better global minimum available.

I believe this happens in a lot of companies, and I believe it's happening in mine.  The new principles document sets goals that implicitly require a lot of stabilization and polishing of our products - we're trying to support customer use cases, but our development so far has been a bleeding edge iterate-as-fast-as-possible mess.

In an ideal world, the smart people in my organization would recognize this, and globally collaborate to focus on the stabilization work that's needed.  However, in the real world, my organization has significant social and financial incentives around creating new features and adding functionality, along with an embedded culture of promoting those things.  There are almost no incentives around stabilization and polishing of products. This results in only new features and functionality being prioritized.  Everything else gets the bare minimum of effort needed to keep our services up, keep things going.

It's a systemic problem across the entire organization, and it's one that can't really be solved at the local level, because of the cultural and financial issues.

This is where our leadership can, and perhaps should step in.  Sometimes, direction and veiled hints just aren't enough.  Sometimes, as a leader, you might want or need to make a hard, concrete tactical decision that actually has consequences if you get it wrong, and use that decision to break a logjam or get to a better global minimum. Elon Musk doing a complete switch from carbon fiber to stainless steel for Starship strikes me as a classic example.

The down side, as a leader, is that it's very easy to get this very wrong.  But if your company is suffering some obvious internal logjam, making a top level tactical decision might be the right thing to do.
# Paper: Large Language Models Can Self-improve [Linkpost]

This paper came out about a week ago. I am not the author - it was published anonymously.

I learned about the paper when JJ Hepburn shared it on Slack. I just thought it seemed potentially really important and I hadn't seen it discussed on this forum yet:

* * *

** Paper title:** "Large Language Models Can Self-improve"

**Author:** Anonymous

**Abstract: "**Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate “high-confidence” rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%→82.1% on GSM8K, 78.2%→83.0% on DROP, 90.0%→94.4% on OpenBookQA, and 63.4%→67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that finetuning on reasoning is critical for self-improvement."

**Link**: [https://openreview.net/forum?id=NiEtU7blzN](https://openreview.net/forum?id=NiEtU7blzN)
# Open & Welcome Thread - Oct 2022

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Against the weirdness heuristic

*Crossposted from the EA Forum:* [*https://forum.effectivealtruism.org/posts/DYmDGizxkcrZhmBCq/against-the-weirdness-heuristic*](https://forum.effectivealtruism.org/posts/DYmDGizxkcrZhmBCq/against-the-weirdness-heuristic)

Slightly more than a decade ago (approximately 2009), I was imagining a pair of earphones that would look exactly like my iPod earphones but would have no wires. That pair was perfect in my imagination: it would fit in my ears just like my wired pair did, the sound would be great, and I wouldn't have to worry about disentangling wires. At the time, wireless meant "ugly Bluetooth (cool) kids never use". In 2016, Apple released the AirPods; they were pretty much what I had imagined plus you needed to charge them (I hadn't thought about that...)

About two decades ago, in the early 2000s, my father went abroad for a few months to work. We knew we wouldn't see his face for that period of time. I don't really remember calling with him either but maybe I'm confabulating here. His absence would have felt way less dramatic were we to video call. In 2010, FaceTime was released while Skype had already been around for many years.

I mention these two stories to argue that both "imagined" technologies became available very soon after I imagined them. I wasn't particularly a tech person, at least no more than the average child growing up at the time. I couldn't have predicted that these technologies should be available based on anything because, well, I didn't know anything. I was just thinking what it'd be nice to have and soon after this thing would be out for sale. (I personally didn't get AirPods till 2022, however).

Thinking about technology in the AI framework often creates the impression that the AGI future is weird. And if it's weird, it's also less likely to happen. This is the weirdness heuristic: I reject something not because I have evidence to do so but because we should generally not anticipate weird things to be true.

In retrospect, it's very easy to assume that having wireless earphones that you charge in a tiny case as well as video calling with someone on the other side of the planet whenever you want are both "normal", non-weird futures. But would we agree on that in the early 2000s?

Writing the "history of the future" would solve many of our problems. We would have clear timelines and plan accordingly. Maybe we would find that there isn't enough time to prepare or even if there is, that we still don't make it. Regardless, the arrow of time doesn't work that way.

All these thoughts lead me to this conclusion: don't resist having short timelines just because "it's too weird" (you may of course have other, legitimate reasons). If, as a kid, you imagined that the day you'd video call with your friends is somewhere in the far distant future and it happened within the same decade, you should expect similar "weird" things to happen in this decade.
# Baby Monitor with Delay

Our youngest (15m) has recently started sleeping through the night on her own, without us needing to get up for feeding or settling. But she's still waking ( [one of](https://www.jefftk.com/p/baby-sleep-multiple-rooms)) us up, because she often cries a bit before falling back asleep. Turning the monitor off entirely is an option, but then if something were pretty wrong (ex: vomiting) we might keep sleeping. I'd be interested in a monitor with a configurable delay: only turn on the monitor if they've been crying for, say, 8min.

A monitor like this would also be useful for sleep training. One common approach is to pick some sort of timing pattern for when to go back in to settle the baby, and not go back in before then. For example, you might settle them, and then hope they stop crying and fall asleep, but if they're still crying in 15min you repeat. Trying to ignore someone you're very attached to cry while trying to remember when exactly it will have been 15min while super sleepy is not fun. If baby monitors had configurable snooze buttons, after settling the baby you could snooze the monitor and try to fall back asleep. If you're lucky they fall back asleep before the fifteen minutes are up, and you more sleep.

There are a bunch of ways to make this fancier, including only counting crying and not other noises or identifying unusual crying, but even something basic like turning on when the average recent volume reaches a threshold would be very helpful.

I did find [someone who built a prototype](https://medium.com/@flagman5/a-baby-monitor-that-delays-notification-3b8a8f17e603) using a phone, with a much shorter delay (10s). I can't find any products, though. Would other people find this useful?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0AbLmMz6Ryg5CB2cpVLrjdopyzisJFwai5gXj7GJnDYxo7pcEVwwZ6BqxDxWPShAQl)*
# Is there a culture overhang?

Culture is adapted for learnability, transmissibility and recombinability for humans. To the extent AI and natural intelligence operate on similar principles, these adaptations should be expected to carry over when culture is used as a training dataset for AI. If so, an AI trained on culture would catch up to the cultural state of the art fast but have more trouble progressing beyond it.  
  
If the above holds, one would expect rapid AI progress up to the frontiers of human culture and nearby low-hanging fruit, after which progress would slow. More speculatively, depending on our compatibility with advancing AI's outputs, humans could have an unexpectedly easy time keeping up with the AI's (relatively slower) acceleration.  
  
Are we currently living in a culture overhang? If so, how does it affect the picture of AI timelines?
# A review of the Bio-Anchors report

This is a linkpost for a review of Ajeya Cotra's [Biological Anchors report](https://www.alignmentforum.org/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines) (see also update [here](https://www.alignmentforum.org/posts/AfH2oPHCApdKicM4m/two-year-update-on-my-personal-ai-timelines)) that I wrote in April 2022. It's since won a prize from the [EA criticism and red-teaming contest](https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/winners-of-the-ea-criticism-and-red-teaming-contest), so I thought it might be good to share here for further discussion.

Here's a summary from the judges of the red-teaming contest:

> This is a summary and critical review of Ajeya Cotra’s [biological anchors report](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP) on AI timelines. It provides an easy-to-understand overview of the main methodology of Cotra’s report. It then examines and challenges central assumptions of the modelling in Cotra’s report. First, the review looks at reasons why we might not expect 2022 architectures to scale to AGI. Second, it raises the point that we don’t know how to specify a space of algorithmic architectures that contains something that could scale to AGI and can be efficiently searched through (inability to specify this could undermine the ability to take the evolutionary anchors from the report as a bound on timelines).

Note that a link to a summary/review of the book [Principles of Deep Learning Theory](https://deeplearningtheory.com/) on page 8 has been moved here: [More Recent Progress in the Theory of Neural Networks](https://www.lesswrong.com/posts/nRu92PXLrdwqdtQmn/more-recent-progress-in-the-theory-of-neural-networks-1).
# If you want to learn technical AI safety, here's a list of AI safety courses, reading lists, and resources

If you're interested in technical AI safety, it's hard to know where to start. Lots of people have made courses and reading lists to help simplify things. 

We [made a spreadsheet of such resources for learning about AI safety](https://docs.google.com/spreadsheets/d/1QSEWjXZuqmG6ORkig84V4sFCldIntyuQj7yq3gkDo0U/edit#gid=0) so you can easily see what's available. It was for internal purposes here at [Nonlinear](http://nonlinear.org/), but we thought it might be helpful to those interested in becoming safety researchers. 

Please [let us know if you notice anything that we’re missing](https://docs.google.com/forms/d/e/1FAIpQLSfRo9ab4Ex-9NzkpTo1nGC6Iz5sWraz5HeAowf7_50wxQ3n3g/viewform?usp=send_form) or that we need to update. This was made about a year ago, but I try to keep it up to date.

Highlights
----------

There are a lot of courses and reading lists out there. If you’re new to the field, out of the ones we investigated, we recommend [Richard Ngo’s curriculum of the AGI safety fundamentals program](https://docs.google.com/document/d/1mTm_sT2YQx3mRXQD6J2xD2QJG1c3kHyvX8kQc_IQ0ns/edit#heading=h.dlm795ug69gc). It is a good mix of shorter, more structured, and more broad than most alternatives. You can [register interest for their program](https://airtable.com/shr41jcUXUc62n1SM) when the next round starts or simply read through the [reading list on your own](https://docs.google.com/document/d/1mTm_sT2YQx3mRXQD6J2xD2QJG1c3kHyvX8kQc_IQ0ns/edit#heading=h.dlm795ug69gc).

The [new one by Dan Hendrycks](https://course.mlsafety.org) also seems great. Structured much more like a classic MOOC and since there are no co-horts, you can just start whenever you'd like.
# Ivermectin: Much Less Than You Needed To Know

I've recently gotten in touch with Alexandros Marinos, which has been writing lengthy critiques of Scott's most famous post, his review of Ivermectin.

After hearing his point of view, I must admit I've been swayed to the thinking that:

1.  There are some obvious statistical errors in Scott's post and his interpretation of Bitterman's epidemiological data
2.  The evidence for ivermectin, while poor, is about as reasonable as that for any repurposed covid drug (e.g. fluvoxamine), and even purpose-made ones (plaxovid)

While I fancy myself an off-label statistician I'm prone to error and maybe my interpretation is wrong. Given that Alexandros has recently finished this post which condenses his core criticisms, I was curious to see what people on LW thought about it.
# No free lunch theorem is irrelevant

I often see people cite the "no free lunch theorem" as a counterargument to the existence of an AGI. I think this is a very bad argument. This theorem is formulated as a problem of **predicting random and uniformly distributed data**. Just open the [Wikipedia](https://en.wikipedia.org/wiki/No_free_lunch_theorem)!. In my opinion, this is another example of how people use results from mathematics simply because they *sound* relevant without understanding their original *meaning* and *limitations*. What makes them think it's relevant to AGI?  
  
It sounds something like “well, since there is such a theorem, it turns out that we will not be able to create an algorithm that can do many tasks.” This is complete nonsense. In order to somehow bind it to real systems, you need to make a reservation and add a restriction on some kind of physical size of the model: the number of parameters, disk space, total equipment weight, training time, total budget whatever. In this case, well, yes. If we have a general algorithm for solving many problems, then for any problem we can probably use the same resources to make an algorithm that performs better. **The problem is that we don't say "how much better"**. Any improvement will do.  
  
It will not be the original theorem, but I think this is close to "what people think when they say no free lunch theorem".  
  
If our algorithm for solving many problems shows itself “kind of meh” compared to specialized algorithms, then what prevents us from increasing the resource budget and collecting all our specialized algorithms into one big one? Yes, the new one can also be reassembled into specialized algorithms, but sooner or later we will hit diminishing returns. It is impossible to calculate arithmetic more accurately than in 100% of cases, it is impossible to predict the structure of a protein more accurately than it will turn out in reality, and so on. **By increasing the available resources**, distilling and amplifying, **we can make a general-purpose algorithm as good as we like in any task**. As close to maximum performance for the metrics as we need.
# Adversarial vs Collaborative Contexts

You can often make pretty clear predictions about what sort of effect something would have in the world as it exists today, but when you start to take into account how the world might change in response it gets very difficult. One tool here is thinking about whether you mostly expect collaborative or adversarial effects.

In a collaborative context other people respond to your change by building on it, and it typically has a larger effect than you'd naively expect. For example, say you create an icon to communicate an idea in your program, other people start using it, and it becomes the standard representation for the concept. While an A/B test might have been only mildly positive, or even negative, after the world adapted to the new icon it would have become very clearly the right symbol to use.

In an adversarial context other people respond to your change by countering it, and it typically has a smaller effect. You figure out a way to [detect bot traffic](https://news.ycombinator.com/item?id=33065604), but the bot operators are very motivated to shift their behavior to undo your work.

Some contexts are also neutral, where both collaborative and adversarial effects are minimal or nonexistent. A baby toy design doesn't get better or worse with the generations.

Some examples:

*   Diseases are adversarial: antibiotics lose resistance, new variants evade immunity. While something looks good in initial testing, that advantage will erode over time. An interesting exception here [is cancer](https://www.jefftk.com/p/cancers-cant-speak), which ([almost always](https://en.wikipedia.org/wiki/Clonally_transmissible_cancer)) has to evolve anew in each patient, and so cancer treatments don't become less effective over time (in new patients).
    
*   Adopting standards is collaborative: each computer with USB increased demand for devices that spoke USB and vice-versa. Someone thinking "let's not bother with USB, it's not that widely supported yet" would have been ignoring collaborative effects.
    
*   Same with building on a platform: the more iPhones sold the more people wanted apps, and the more apps there were the more people wanted iPhones.
    
*   Interaction with nature is generally adversarial: cockatoos figure out how to [get into trash](https://www.sciencedirect.com/science/article/abs/pii/S0960982222012854), weeds evolve herbicide resistance.
    
*   A new kind of ad has both kinds of effects: adversarial in that people will notice novel ads more, but cooperative in that advertisers haven't yet learned how to make the most of the format.
    

This sort of effect is especially important to think through when considering rolling something out based on the results of an experiment. You've almost certainly not run the experiment long enough to see how the world will adjust, but you can use these patterns to predict whether the long-term results will be stronger or weaker than you've measured.
# Paper+Summary: OMNIGROK: GROKKING BEYOND ALGORITHMIC DATA

There is a new paper by Liu et al. that claims to have understood the key mechanism underlying grokking (potentially even generalization more broadly). 

  
**They argue: **  
1\. Grokking can be explained via the norm of the weights. They claim that there is a constant level of the weight norm that is optimal for generalization.  
2\. If there is an optimal level of the weight norm, the weight norm of your model after initialization can be either too low, too high or optimal. They claim that grokking is a phenomenon where we initialize the model with a large weight norm and it then slowly walks toward the optimal weight norm and then generalizes. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0a558d26a6b84fa9e752f064dcc5e2e4c5d20a2d3c87b18a.png)

3\. They also claim that you can get the same results as grokking but much faster if you set the weight norm correctly at every step.  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/59d5e5d9145319ee7d66cb434ad5ee993cec11371c1f5744.png)  
4\. They set the norm "correctly" by rescaling the weights after each unconstrained optimization step (so after every weight update loss.backward()?!)  
  
**Implications:**

1.  I think they have found a very important insight of grokking and finding generalizing circuits more broadly. 
2.  I'm still a bit skeptical of some of the claims and results. On some level "just fix your weight norm and the model generalizes" sounds too simple to be true for all tasks. 
3.  I think this result could have big implications but I'm not yet sure whether they are positive or negative. On the one hand, finding generalizing circuits seems to solve some of the problems associated with bad out-of-distribution generalization. On the other hand, it likely speeds up capabilities. 

I'm very unsure about this paper but intuitively it feels important. Thoughts?
# Does Google still hire people via their foobar challenge?

In years since past, the [foobar challenge](https://towardsdatascience.com/google-foobar-challenge-level-1-3487bb252780) first appeared. 5 levels of ever harder coding challenges, and if you bested the first three you'd be given the option of providing your details to Google. There was a decent chance you'd be sent an interview request sometime later or , if you applied to them yourself, you're application would be given extra weight. Recently, I got the challenge from Google and after getting to the last problem of level three, I wondered "do they even hire people using this thing anymore"? Some quick searches on medium blogs, quora and hackernews later, I was left with the impression that they don't but no one could cite a source for this. 

So does anyone have any info, definitive or otherwise, that would tell me and other lost souls whether Google still hires using this thing? 

Edit: changed the name
# Feature request: Filter by read/ upvoted

I am looking for a post that I read and probably upvoted. I can't find it. I am 90% sure you know which posts I've read and 99.9% sure you know which posts I've upvoted. Can I be able to filter by them.  
  
That's it. That's the post.

It's that post about not having to have perfect views but just saying why you have the views you have so you are easy to correct.
# Paper: Discovering novel algorithms with AlphaTensor [Deepmind]

The authors apply an AlphaZero-like algorithm to discover new matrix multiplication algorithms. They do this by turning matrix multiplication into a one-player game, where the state represents how far from correct the current output is, moves are algorithmic instructions, and the reward is -1 per step (plus a terminal reward of -rank(final state), if the final state is not a zero tensor). On small matrices, they find that AlphaTensor can discover algorithms that use fewer scalar multiplications than the best known human-designed matrix multiplication algorithms. They apply this to find hardware-specific matmuls (by adding an additional reward equal to -time to the terminal state) that have a 10-20% larger speedup than Strassen's algorithm on NVIDIA V100s and TPU V2s (saving 4%/7.5% wall clock time). 

Paper abstract:
> Improving the efficiency of algorithms for fundamental computations can have a widespread impact, as it can affect the overall speed of a large amount of computations. Matrix multiplication is one such primitive task, occurring in many systems—from neural networks to scientific computing routines. The automatic discovery of algorithms using machine learning offers the prospect of reaching beyond human intuition and outperforming the current best human-designed algorithms. However, automating the algorithm discovery procedure is intricate, as the space of possible algorithms is enormous. Here we report a deep reinforcement learning approach based on AlphaZero for discovering efficient and provably correct algorithms for the multiplication of arbitrary matrices. Our agent, AlphaTensor, is trained to play a single-player game where the objective is finding tensor decompositions within a finite factor space. AlphaTensor discovered algorithms that outperform the state-of-the-art complexity for many matrix sizes. Particularly relevant is the case of 4 × 4 matrices in a finite field, where AlphaTensor’s algorithm improves on Strassen’s two-level algorithm for the first time, to our knowledge, since its discovery 50 years ago. We further showcase the flexibility of AlphaTensor through different use-cases: algorithms with state-of-the-art complexity for structured matrix multiplication and improved practical efficiency by optimizing matrix multiplication for runtime on specific hardware. Our results highlight AlphaTensor’s ability to accelerate the process of algorithmic discovery on a range of problems, and to optimize for different criteria.
# Tracking Compute Stocks and Flows: Case Studies?

*Posted in my personal capacity*

The AGI governance community has recently converged on compute governance^[See, e.g., [this sequence](https://forum.effectivealtruism.org/s/4yLbeJ33fYrwnfDev) by Lennart Heim.] as a promising lever for reducing existential risks from AI. 

One likely building block for any maximally secure compute governance regime is **stock and flow accounting of (some kinds of) compute**: i.e., requiring realtime accurate declaration to regulators of who possesses which uniquely numbered regulated chips, with penalties for undeclared or unauthorized^[The question of which types of transfers ought to be authorized is important but beyond the scope of this post.] transfers. 

To understand the optimal design and feasibility of such a regime, we seek historical analogies for similar regimes. One that we are already familiar with include:

* Fissile nuclear material and other nuclear weapons components
* Firearms
* Some financial instruments
* Automobiles
* Real estate

**What are other good existing or historical analogies for compute stock and flow accounting**? An ideal analogy will have many of the following traits:^[NB: Many of the above do not have all of these traits!]

* The thing being tracked is a physical object
* The thing being tracked is economically important
* The thing being tracked is dual-use
* The tracking regime requires registration of current ownership and any transfers
* The tracking regime imposes penalties for failing to register ownership or transfer

More and better examples of stock and flow accounting mechanisms could be extremely informative in improving compute governance proposals.
# Finding Great Tutors

Does anyone have tips for finding great tutors for technical subjects? I'm looking to speed up my self-studying for eventual alignment research. Money isn't much of an issue, I'd probably meet every few days to talk about stuff that confused me and generally grill my teacher. :)

I'm mostly studying topics from John's [Study Guide](https://www.lesswrong.com/posts/bjjbp5i5G8bekJuxv/study-guide), so calibrate roughly for that kind of content.
# Do uncertainty/planning costs make convex hulls unrealistic?

It's almost a rule that as soon as you have a "utility of possible outcomes" plot like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b87197cae9a5a70e19e9112273f0082bbf2227b0d7d40395.png)

You must then say "and by randomly choosing between the outcomes, we can achieve any intermediate outcome in terms of utility within the convex hull of these points" resulting in a plot like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2af0075d59e4e068dfe0000787010d355eec8f6927dc966f.png)

Cool, I've done a linear interpolation before, seems reasonable. Plus, convex hulls are super nice to work with. But all models are imperfect - how accurate is this convex hull idea in practice?

Three stories
-------------

*   I like to plan ahead, whereas my friend values chaos and unpredictability. When we want to go to a restaurant, they're a big fan of the whole "lottery between your first preference and my first preference" idea.
    *   If we're meeting at 7PM and the lottery happens at 7AM that's long enough to plan, so I don't mind.
    *   If we're meeting at 7PM and the lottery happens at 6:59PM, I am a little annoyed - I might even prefer my guaranteed second preference over my uncertain first preference.
    *   The "standard model" for what the outcome space looks like before picking $p=\mathbb{P}(\mathrm{my preference})$ is something like this:
    *   ![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1332d11cbe21abb0b820f048517e967755fa4bc47ed6a7f2.png)
*   I'm a CEO managing a business undergoing a possible merger. The more uncertain the deal is, the more all of my projects are disrupted. I might be fine walking into the final meeting with a 99% or 1% chance of success, but I would be quite stressed if the numbers were 90% or 10%.
*   I'm a computer, beep boop. I've been modelling deterministic systems all day long. Someone has just asked me to model something probabilistic, and now I need to learn around 70 years of research to not be overwhelmed by the state space explosion.

My point is that in practice the mapping from lottery probability $p$ and outcome utilities $U_1,U_2$ to lottery utility $U_p$ is probably not $U_p=pU_1+(1-p)U_2$. I wonder if it's occasionally *not even close*. I would expect the "lottery closure" of the three outcomes above to look something like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a02b7db5ed27bede8108e2be329297701a14afb26bec5090.png)

I'm pretty darn sure I'm not the first person to think about this, so: how big an issue is this when thinking about things like bargaining solutions, or stochastic Nash equilibria?
# linkpost: neuro-symbolic hybrid ai

This video shows some really interesting work that's been done on factoring sensory perceptions (visual data, natural language) into concept space (they call it 'derendering' as an antonym to rendering), and then applying symbolic reasoning to the concepts. I think this is really potentially powerful for data efficiency and interpretability. As with many things that I and others post about, I don't think this is a notion sufficient for solving alignment on its own, but I do think human-readable symbolic reasoning is an important piece that I'd be surprised to learn that future alignment solutions didn't include.

Edit: for the tldr version: watch from 23:50 to 28:20
# Calibrate - New Chrome Extension for hiding numbers so you can guess

One thing I wish I did more often is checks to see if I understand the world - Fermis, landmark numbers, predictions about how things work, etc.

Along the way I started wishing I could cover up numbers on websites so I would have the chance to guess first and then reveal via mouseover, and [Aur Saraf](https://www.facebook.com/aursaraf?__cft__[0]=AZWH2acpWIvLUUEVThk6Y158wM2tWOi9rjgrPPOql6posLbMyiTTlfGco38LAqF5fAti7k9Q5Hdt92IJc5BoFeIHK8nU6paELrzz5d9WE8aNdYmX9CO9HbSOKlpPLMSQZgQ&__tn__=-]K-R) agreed to make a chrome extension, and [here it is!](https://chrome.google.com/webstore/detail/calibrate/omhpddekmhnkmpbjogkfeihglpjdndhf/related?fbclid=IwAR09kbuaP6wAy-hiJrQpwQsR1C9DJLggY6QzypPbX_1uTtLq193DNwRU98o) Lets you set the default for a whole site or a page. (Not good for dynamic sites like twitter)

Extremely excited.

The code is open source [here](https://github.com/SonOfLilit/calibrate?fbclid=IwAR2vBZ8IWfMgHTPla0CbohCUIqmrMUl-XEcYIWhKUrJ4ZRfH2Eg7Z7Zf1J4) if you want to add any feature requests.

(I think I want to change the name - let me know if you have thoughts)  
  
![](https://lh3.googleusercontent.com/gWC6NNXjO3CT7-nVBb6gbcpsbnWDhpiFvGVTnGpqn9J-egpCAJNEa2daWs4Rt8Q2oCDLA6fpjzM8s4QoickBJOvIjg=w640-h400-e365-rj-sc0x00ffffff)
# Deliberate practice for research?

I would like to gain mastery in the domain of alignment research. Deliberate practice is a powerful sledge hammer for gaining mastery. But unlike something like chess or piano, it's not clear to me how to use this sledge hammer for this domain. The feedback loops are extremely long, and the "correct action" is almost never known ahead of time or even right after doing the action.

**What are some concrete ways I could apply deliberate practice to alignment research?**

One way would be to apply it to skills that are sub-components of research, rather than trying to rapidly practice research end-to-end.

The sub-skill I've thought of that is the best fit to deliberate practice is solving math and physics problems, a la Thinking Physics or other textbook exercises. Being better at this would certainly make me a better researcher, but it might not be worth the opportunity cost, and if I ask myself, "Is this cutting the enemy with every strike?" then I get back a no.

Another thing I can think of is trying to deliberately practice writing, which is a big part of my research. I could try to be more like John, and write a post every week, to get lots of quick feedback. But is this fast enough for deliberate practice? I get the sense that the feedback cycle has to be almost real-time. Maybe doing tweet explanations is the minimal version of this?

I'd appreciate any other concrete ideas! (Note that my research style is much more mathy/agent-foundations flavored, so programming is not really a sub-skill of my research.)
# The "you-can-just" alarm

Daniel Dennett argues for the idea of a "[Surely alarm](https://medium.com/science-and-technology/how-to-spot-a-weak-argument-83dacb1fe14c)" which says that whenever you hear someone say "surely" in an argument, an alarm should sound in your head—\*ding!\*—and you should pay attention for weaknesses. When someone says "surely", it's often just meant to appeal to a sense of *disbelief* that anyone could possibly disagree with them.

Readers of the Sequences will be familiar with similar alarms upon hearing "[emergence](https://www.lesswrong.com/posts/8QzZKw9WHRxjR4948/the-futility-of-emergence)", "[by definition](https://www.lesswrong.com/posts/cFzC996D7Jjds3vS9/arguing-by-definition)", "[scientific](https://www.lesswrong.com/posts/L22jhyY9ocXQNLqyE/science-as-curiosity-stopper)", "[rational](https://www.lesswrong.com/posts/hN8Ld8YdqFsui2xgc/only-say-rational-when-you-can-t-eliminate-the-word)", "[merely](https://www.lesswrong.com/posts/x4dG4GhpZH2hgz59x/joy-in-the-merely-real)", and more. And if you're *really* unlucky, some words just [explode](https://www.lesswrong.com/posts/L4HQ3gnSrBETRdcGu/philosophical-landmines) without warning.

Anyway, when you notice that people aren't doing something to attain what they've explicitly expressed a preference for, you should be wondering what's going on. I've seen countless examples of people noticing the discrepancy only to brush it aside because *"surely if they wanted to do it, they would have."* People routinely underestimate the cost of some [seemingly insignificant](https://www.lesswrong.com/posts/9kNxhKWvixtKW5anS/you-are-not-measuring-what-you-think-you-are-measuring) obstacles.

The fact that ***A)*** **it's obvious that people really want Y**, and that ***B)*** **it's also obvious that there's almost no cost to doing X in order to attain Y**, should not just cause you notice-it-as-plain-fact. There's a deep mystery here, and if *you also* believe both A and B, there's real epistemic profit to be had by paying close attention. The potential profit is proportional to the total strength of the beliefs you've just noticed are in direct conflict.

You might naively think that if the confusion is so profitable, surely you—with all your ability to reason and pay close attention to reality—would've already noticed and exploited it. But if acquired beliefs A and B from a pre-attentive stage of your life, this may also explain why you've skipped over this obvious opportunity until now.

Besides, it's really cheap to do a second look-over, so *why-don't-you-just* *investigate*?
# How many GPUs does NVIDIA make?

There are a few subquestions that I've had trouble finding numbers on with a quick search. Asking these questions mostly because they seem important for forecasting compute trends.

*   How many units of each model (i.e A100, 3090, etc) does NVIDIA make per month? 
*   Which of these use the same dies but have constrained supply ratios due to binning? What do these ratios look like/can they change if NVIDIA decides to focus on high end GPUs?
*   How much of the total global silicon capacity at the latest process node does this take up? How hard would it be for NVIDIA to scale up by squeezing out other silicon usages?
# Signaling Guilt

Does anyone feel "signaling guilt"?

When I have to write something like a CV/Resume/essay for the purpose of admission into a high-status institution (e.g., academia, high-paying job), I feel an insane amount of [ugh-field](https://www.lesswrong.com/posts/EFQ3F6kmt4WHXRqik/ugh-fields) and find myself unable to properly motivate myself into actually focusing on it.

All of it just feels ... *dishonest.*

It gets worse with more competitive positions; just good ol' [Moloch](https://www.lesswrong.com/tag/moloch) doing what it's good at: race-to-the-bottom, throwing away everything of value for the sake of slightly increasing competitiveness at the margin, etc.

Every time when I have to (fake intentions and emotions / intentionally distort my value system / signal fake compliance / pander to the reviewers by adopting their political stance and lingo / do something solely for the sake of signaling), it feels like I'm corrupting my "*soul"* via some irreversible process.

Perhaps the concept of "signaling" is [infohazardous](https://www.lesswrong.com/tag/information-hazards)—I wasn't like this a couple years ago, back when I unconsciously optimized *everything* for the sole purpose of "appearing good in my resume," completely guilt-free.

Not any more. And no one around me seems to get it.

These days I find myself ... just *baffled,* as to how my colleagues can even *do* what they're doing while staying sane. *Entire lives*, lived without a single shred of *real* substance, just pure, *pure* signaling. I *read* the [Moral mazes](https://www.lesswrong.com/tag/moral-mazes) and understand how institutions select for *lizards* not actual living-breathing-humans—

And I tried so many things—artificially pumping my short-term motivation with stimulants, outsourcing my writings to GPT-3, trying to compartmentalize my life—none of them work consistently, at least for me.

The only thing that's keeping me going at this point is the tiny, *tiny* positive-reinforcment from the (false) belief that "I am a *rare* instance of an *actual living-breathing-human* who is *heroically* resisting the selection pressures and fighting against the *Evil* and *Immoral* *Lizards.*"

Not for so long. (did I mention that I'm actually supposed to be editing my resume right now?)

Perhaps people actually *do* feel these signaling guilt but somehow manage to pull themselves together.

Perhaps this is just me conflating my normal akrasia and blaming it on The System.

Perhaps I'm just not Machiavellian enough.  
In that case, well, *too bad!*