Harley-ml/lesswrong huggingface dataset:
# Opening Facebook Links Externally

When I'm on my phone and click a link in Facebook I want it to open in my normal browser, but it defaults to an "in-app" browser. I really don't like in-app browsers, and I think [Alex Russell's explanation](https://infrequently.org/2021/07/hobsons-browser/) of their problems is very good. I've turned this behavior off, but Facebook regularly forgets my decision, most recently a few weeks ago. The setting had moved since [the last time I did this](https://www.jefftk.com/p/selecting-the-default-browser-in-facebook-on-android), so here are screenshots showing how to turn it off on Android:

Click on the icon of your profile image overlaid with a hamburger menu in the upper right, and scroll to the bottom:

[![](https://www.jefftk.com/fb-noiab-1.jpg)](https://www.jefftk.com/fb-noiab-1-big.jpg)

Click "Settings & privacy":

[![](https://www.jefftk.com/fb-noiab-2.jpg)](https://www.jefftk.com/fb-noiab-2-big.jpg)

Click "Settings":

[![](https://www.jefftk.com/fb-noiab-3.jpg)](https://www.jefftk.com/fb-noiab-3-big.jpg)

Click "Profile settings":

[![](https://www.jefftk.com/fb-noiab-4.jpg)](https://www.jefftk.com/fb-noiab-4-big.jpg)

Click "Media and contacts" settings:

[![](https://www.jefftk.com/fb-noiab-5.jpg)](https://www.jefftk.com/fb-noiab-5-big.jpg)

Toggle "Links open externally":

[![](https://www.jefftk.com/fb-noiab-6.jpg)](https://www.jefftk.com/fb-noiab-6-big.jpg)
# Infohazards vs Fork Hazards

I think actual infohazardous information is fairly rare. Far more common is a fork: you have some idea or statement, you don't know whether it's true or false (typically leaning false), and you kow that *either* it's false *or* it's infohazardous. Examples include unvalidated insights about how to build dangerous technologies, and most acausal trade/acausal blackmail scenarios. Phrased slightly differently: "infohazardous if true".

If something is wrong/false, it's at least mildly bad to spread/talk about it. (With some exceptions; wrong ideas can sometimes inspire better ones, maybe you want fake nuclear weapon designs to trip up would-be designers, etc). And if something is infohazardous, it's bad to spread/talk about it, for an entirely different reason. Taken together, these form a disjunctive argument for not spreading the information.

I think this trips people up when they see how others relate to things that are infohazardous-if-true. When something is infohazardous-if-true (but probably false), peopple bias towards treating it as actually-infohazardous; after all,if it's *false*, there's not much upside in spreading bullshit. Other people seeing this get confused, and think it's actually infohazardous, or think it isn't but that the first person thinks it is (and therefore thinks the first person is foolish).

I think this is pretty easily fixed with a slight terminology tweak: simply call thinks "infohazardous if true" rather than "infohazardous" (adjective form), and call them "fork hazards" rather that "infohazards" (noun form). This clarifies that you only believe the conditional, and not the underlying statement.
# PSA: reward is part of the habit loop too

The usual setup of a habit is

1.  cue
2.  craving
3.  routine
4.  _**reward**_

Among my social circle, rewarding oneself seems to be lost by the wayside. Sadly funny for a crowd of people into reinforcement learning, since they're trying to skip the reinforcement and wondering why it doesn't stick^[1](#fn1)^. My reward is usually reading fiction or playing a video game. For brushing my teeth, it's the nice shiny feeling at the end, which I make a point of noticing. For going to the gym, it's the steam room after.

Figure out what you like. Anything can be a reward as long as it _feels_ good. Don't try to logic yourself into wanting "the right rewards". Has that worked before? Surrender. Let the soft animal of your body love what it loves.

1.  RL without the reward is an _environment_ (states, actions, transitions) that's Markovian. It has a syntax (things are happening), but no semantics (without a reward function, there's no interpretation of whether something is good or bad). [↩](#fnref1)
# Running by Default

In video games it's common for characters to [run by default](https://tvtropes.org/pmwiki/pmwiki.php/Main/RunDontWalk): it's faster, why wouldn't you? I think this is also a good approach to apply in real life, at least over short distances. It saves you time and it's a bit of exercise. Over pretty much any distance short enough that I'm not going to break a sweat I'll run.

I've done some of this most of my life (I can still hear my middle school principal shouting to stop running in the hallway—*Jeeeeff-reeeey!*) though over the last five years or so I've pushed myself to do it [more](https://www.jefftk.com/p/running-and-optimizing) [consistently](https://www.jefftk.com/p/weight-strategies). The more I do it the easier it gets, and the shorter commutes become.

Note that this is [another example](https://www.jefftk.com/p/preparing-for-less-privacy) of describing something I do that works for me, and that I expect would likely work for many more people than I see applying it. But probably not more than 30% of people, though?

[![](https://www.jefftk.com/jeff-kid-running.jpg)](https://www.jefftk.com/jeff-kid-running-big.jpg)

(Prompted by a coworker talking about the inconvenience of an 8min distance between two buildings at work that I think of as a 2-3min distance.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02Rmqz1qBoAfoeedbsGDjanBCxD8SdMoBmjyTuDwSGZZKBakghBTPi4PZ2Qj19txj1l), [mastodon](https://mastodon.mit.edu/@jefftk/109636909414313777)*
# Machine Learning vs Differential Privacy

edit: see below for clarifications by  expert domain rpglover64 and a good pick of references from the gears to ascenscion. My one (cheatingly long) sentence takeaway is: it’s clear training does not automatically lead to DP, it’s unclear if DP can always or seldom help training, it’s likely that easy algorithms are not available yet, it’s unlikely that finding one is low hanging fruit.

From wikipedia, « an algorithm is differentially private if an observer seeing its output cannot tell if a particular individual's information was used in the computation ». In other words, if some training process asymptotically converges toward generalisable knowledge only, then it should tend to become differentially private.

…or so it seems to me, but actually I’ve no idea if that’s common knowledge in ml- or crypto- educated folks, versus it’s pure personal guess and there’s no reason to believe that. What do you see as the best argument for or against that idea? Any guess on how to disprove or prove it?

Extra Good Samaritan point : my english sucks, so any comment rewriting this post in good english, even if for minor details, is a great help thank you.

This is version 0.1
# Paper: Superposition, Memorization, and Double Descent (Anthropic)

(This is a follow up to Anthropic's prior work on [Toy Models of Superposition](https://arxiv.org/abs/2209.10652).)

The authors study how neural networks interpolate between memorization and generalization in the "ReLU Output" toy model from the first toy model paper:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1b9cf40d8eb3e692fe9a2e85fa4bdd683a768462a8d4a1b7.png)

They train models to perform a synthetic regression task with $T$ training points, for models with $m$ hidden dimensions. 

First, they find that for small training sets, while the features are messy, the training set hidden vectors $h_i​=WX_i​$ (the projection of the input datapoints into the hidden space) often show clean structures:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e4a5db2e098ee91b40da9687a52cdc2f5cf0e9cae523a5a4.png)

They then extend their old definition of feature dimensionality to measure the dimensionalities allocated to each of the training examples:

$D_{X_i} ~=~ \frac{||h_i||^2}{\sum_j (\hat{h_i} \cdot h_j)^2}$  
and plot this against the data set size (and also test loss):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/23598ae2265b7cd222ef0f8403147129ea49562c87c81e6b.png)

This shows that as you increase the amount of data, you go from a regime with high dimensionality allocated to training vectors and low dimensionality allocated to features, to one where the opposite is true. In between the two, *both* feature and hidden vector dimensionalities receive low dimensionality, which coincides with an increase in test loss, which they compare to the phenomena of "data double descent" (where as you increase data on overparameterized models with small amounts of regularization, test loss can go up before it goes down).

Finally, they visualize how varying $T$ and $m$ affects test loss, and find double descent along both dimensions:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/df1d0b41e4780a1724d6cb0dd7496d58e7a62d41e90dd652.png)

They also included some (imo very interesting) experiments from Adam Jermyn, 1) replicating the results, 2) exploring how weight decay interacts with this double descent-like phenomenon, and 3) studying what happens if you repeat particular datapoints.

* * *

Some limitations of the work, based on my first read through:

*   The authors note that the results seem quite sensitive to hyperparameters, especially for low hidden dimension $m \in \{2, 3\}$. For example, Adam Jermyn's results differ from the Anthropic Interp team results (though the figures still look qualitatively similar).
*   I'm still not super convinced how much results from the superposition work apply in practice. I'd be interested in seeing more work along the lines of the MNIST preliminary experiment done by Chris Olah at the bottom. 

(I'll probably have more thoughts as I think for longer.)
# How to slow down scientific progress, according to Leo Szilard

Leo Szilard—the physicist who first conceived of the nuclear chain reaction and who urged the US to undertake the Manhattan Project—also wrote fiction. His book of short stories, *The Voice of the Dolphins*, contains a story “The Mark Gable Foundation,” dated 1948, from which I will present to you an excerpt, without comment:

> “I’m thinking of setting up a trust fund. I want to do something that will really contribute to the happiness of mankind; but it’s very difficult to know what to do with money. When Mr. Rosenblatt told me that you’d be here tonight I asked the mayor to invite me. I certainly would value your advice.”
> 
> “Would you intend to do anything for the advancement of science?” I asked.
> 
> “No,” Mark Gable said. “I believe scientific progress is too fast as it is.”
> 
> “I share your feeling about this point,” I said with the fervor of conviction, “but then why not do something about the retardation of scientific progress?”
> 
> “That I would very much like to do,” Mark Gable said, “but how do I go about it?”
> 
> “Well,” I said, “I think that shouldn’t be very difficult. As a matter of fact, I think it would be quite easy. You could set up a foundation, with an annual endowment of thirty million dollars. Research workers in need of funds could apply for grants, if they could make out a convincing case. Have ten committees, each composed of twelve scientists, appointed to pass on these applications. Take the most active scientists out of the laboratory and make them members of these committees. And the very best men in the field should be appointed as chairmen at salaries of fifty thousand dollars each. Also have about twenty prizes of one hundred thousand dollars each for the best scientific papers of the year. This is just about all you would have to do. Your lawyers could easily prepare a charter for the foundation. As a matter of fact, any of the National Science Foundation bills which were introduced in the Seventy-ninth and Eightieth Congresses could perfectly well serve as a model.”
> 
> “I think you had better explain to Mr. Gable why this foundation would in fact retard the progress of science,” said a bespectacled young man sitting at the far end of the table, whose name I didn’t get at the time of introduction.
> 
> “It should be obvious,” I said. “First of all, the best scientists would be removed from their laboratories and kept busy on committees passing on applications for funds. Secondly, the scientific workers in need of funds would concentrate on problems which were considered promising and were pretty certain to lead to publishable results. For a few years there might be a great increase in scientific output; but by going after the obvious, pretty soon science would dry out. Science would become something like a parlor game. Some things would be considered interesting, others not. There would be fashions. Those who followed the fashion would get grants. Those who wouldn’t would not, and pretty soon they would learn to follow the fashion, too.”
# Get an Electric Toothbrush. 

Okay so, with the inspirations of the concise posts about lifestyle that user @jefftk does, I want to share one of my most recent discoveries with LW, one that with little economical input can offer you a better quality of life. 

*GET AN ELECTRIC TOOTHBRUSH!!* 
===============================

Really! A 20 bucks Oral-B department store is fine! 

Do you remember the nice feeling when you go to your dentist for a cleanup and you leave with that smooth, polished feeling on your teeth that sometimes last you days? I do remember it, I got the exact same feeling from my first brush with an Electric toothbrush. 

I got a Basic Oral-B Vitality[^nsnt8lebjam] from my grocery store a week ago as a suggestion from my girlfriend, and is the best palpable investment of the year for me. 

You don't need anything fancy. It doesn't have to be a big investment like the 100$ plus ones with pressure sensors, a basic, 2 minute timer electric toothbrush will get you 90% of the way there, a YouTube video tutorial about how to use them effectively, another 5% 

### Stuff that's good to know. 

*   You need to touch the gums with the toothbrush
*   Almost no pressure is necessary, I hold it with two fingers to remind myself, the rapid oscillation of the head will do the rest
*   Just move it from teeth to teeth, no more "toothbrush" movement required. 

After a week of use, I feel confident enough to share the news. it's the superior way of cleaning your teeth, although I miss the caveman feeling of nuking my teeth with a hard brush from time to time. 

[^nsnt8lebjam]: This is European branding, I'm almost certain you can find the same product in the US or other markets from Oral-B with different names, a quick search gives me This stuff but I'm sure you can find something similar on your local Walmart
# The Involuntary Pacifists

This might be an idea better put to r/WritingPrompts (or ChatGPT), but a scenario has occurred to me which might count either as a near-utopia, or extreme dystopia, depending on your point of view.

Imagine a future Earth where a great power (this can be a superintelligent AI, benevolent alien race, or just straight-up God, depending on your preferences) has imposed a version of the libertarian [harm principle](https://en.wikipedia.org/wiki/Harm_principle) on humanity. Most actions a human can engage in today would be available to you, you just wouldn't be able to commit an act of violence against another human. Sure, you might have an extremely strong *desire* to harm a human, but try as you might, your brain just can't send the signals to the rest of your body in order to pull that trigger/prepare that device/program that killbot.

We should also define "harm" very narrowly here. A person scamming an elderly lady out of her life savings is definitely harm, so is hurling racial epithets at someone, but I think we should here reserve the word for the most serious of crimes. These involve a violation of bodily integrity. Murder, assault, rape, and forcible confinement. What would this world where you can't do violence to a fellow human be like? Getting your head bashed in with a tire iron during a road rage incident would be something you read about in historical fiction. Same with war. You can gather all the men and material you possible can, but you simply wouldn't be able to use them. Seems pretty utopian to me, though I'm aware [sometimes important details are hidden from cursory glances at a possible world](https://www.lesswrong.com/posts/nFv2buafNc9jSaxAH/siren-worlds-and-the-perils-of-over-optimised-search). A possible problem is how do you punish that guy who scammed the grandma? Taking him into custody would necessarily involve acts of violence no longer available to humans.

We could even limit this to non-consensual violence. All you MMA types, having formally agreed to accept violence from an opponent, would be free to punch each other's faces in. You could even have voluntary wars, where all the people who really miss the sound and fury of battle can meet up in central Australia or somewhere and have at it, leaving the rest of humanity in peace.

It would be interesting to see how conflict among humans evolves under these conditions. Two primitive tribes are competing for a scarce water source. So one tribe comes to take it from the other by force, and then I guess they stand around with constipated expressions before giving up and going home. But that means the tribe already there gets the precious water resource just by having got there first. Is that fair?

I also don't see this as a violation of free will, as you would still have the desire to harm, just not the freedom to act on it. Same way a prisoner, though confined, has the overarching desire to escape at the first opportunity.

So what detail would reveal this prospective utopia to be, in fact, a hellish dystopia?
# Ball Square Station and Ridership Maximization

Matt Yglesias recently wrote [American transit agencies should prioritize ridership over other goals](https://www.slowboring.com/p/american-transit-agencies-should), and described how Boston's Green Line Extension was:

> so far over budget that it got canceled and then re-launched in a cheaper way. One key to reducing costs was eliminating plans to build large custom stations and going with simple bare-bones ones instead. So why did they propose the large stations in the first place? I have no idea. The existing and well-used Green Line stations are incredibly bare-bones. There was nothing in the local transit tradition to suggest that the extension needed large grand ones. All else being equal, people might prefer a really grand station to a bare-bones one, but there was no MBTA modeling that claimed this expenditure would make the system dramatically more appealing. They simply weren't asking the question "what will the ridership impact of this spending be?" because American transit agencies aren't in the habit of asking those questions.

Yglesias is citing a [Transit Costs Project report](https://transitcosts.com/wp-content/uploads/Boston-Case_The-Story-of-the-Green-Line-Extension.pdf), and his summary is fair. In fact, the 2016 redesign wasn't just more riders per dollar but, according to the MBTA, the same number of projected riders for a much lower cost (p283-284 of the 2019 [TIP report](https://www.ctps.org/data/pdf/plans/TIP/FFYs-2020-2024-Final-TIP-20190730.pdf)). But Yglesias (and the [TCP](https://transitcosts.com/) report) imply that the stations as finally built are simpler and cheaper structures than they actually are.

Let's look at the Ball Square station, the one I'm most familiar with since it's close to my house. Here's the [layout](https://www.mass.gov/doc/ball-square-station-design/download):

[![](https://www.jefftk.com/ball-square-station-design-top.png)](https://www.jefftk.com/ball-square-station-design-top-big.png)

[![](https://www.jefftk.com/ball-square-station-design-side.png)](https://www.jefftk.com/ball-square-station-design-side-big.png)

Let's look specifically at the second entrance. It required an elevator, stairs, a three-story structure with a roof, and a connection to an existing bridge. What do you get? For riders heading east across the Broadway bridge it saves ~150ft horizontally (140ft vs 290ft) and ~20ft vertically. This probably applies to ~1/3 of riders, based on the fraction of nearby housing is across the bridge and that Ball Square itself is on the station side of the bridge.

I can see an argument that this entrance was worth the cost, by making it easier and faster for people who need to cross the bridge, especially if they use wheelchairs or strollers. Alternatively, keeping the money and using it to help extend the Green Line [to Route 16](https://en.wikipedia.org/wiki/Mystic_Valley_Parkway_station) might have been better, or spending it elsewhere in the system. But either way it's an example of the kind of thing I think you wouldn't build if you were following Yglesias' proposal to maximize ridership alone.
# AI security might be helpful for AI alignment

I am a person new to AI alignment, and recently I am looking for a job at the AI governance field, but it seems like it is hard to find a long-term financially stable full-time job. 

For me, it basically means that only a tiny amount of people consider AI alignment important enough to pay money for the decrease of P(doom).

It is important to get more people involved in the AI alignment field, but there is another good way to increase the number of people who might help achieve similar goals. We might cooperate with other groups of people, whose agenda is beneficial for AI alignment. One such group is the AI security community.

  
 

### **AI security**

AI security is a branch of software security. Its goal is to ensure that current AI systems can not be exploited by hackers, are free of backdoors, and generally do things they are intended to do and do not do any unexpected and potentially harmful things.

For example, there is a grocery store not far from my home, which uses cameras with facial recognition algorithms to identify a buyer and automatically charge money from their credit card. Malicious actors might use some kind of makeup or a mask to trick the algorithm. So, to prevent it, the operator of this payment system might hire an AI security engineer to simulate an attack on the model, find its vulnerabilities and fix them.

Why this company should care that its facial recognition system has no security issues? Because if someone decides to hack it and steal money from its customer, it might become a huge scandal and lead to lawsuits, so it is relatively easy to convince top management to pay money for AI security. For them, it is just another branch of software security.

A similar thing with governments. I think it is manageable to convince governments to implement policies and guidelines for safe and robust AI systems because we already have similar policies for other important software.

AI security has a lot in common with the task of aligning AGI. There are few of them: 

*   It benefits from interpretable models. 
*   It requires the education of major decision-makers about the risks of unsafe AI systems.
*   Government and enterprise policies might require thorough testing and certification of AI models, it not only will make systems safer, but also might slow down the rate of AI progress which is good for alignment. 
*   AI security also requires systematic audit of AI models, some kind of alarm system in case of an attack, and for containment of its consequences.

AI security researchers and organizations might help with exposing AI-associated risks to the general public. They might develop new secure and robust AI systems, they might become allies in policymaking.

I further think that as technology progresses, AI security's goal of making sure that important and powerful AIs are doing what they are intended to do and do no harm to people will converge with the AI alignment goal of making sure that AGI will do what it is intended to do and not cause no harm to people.
# Open & Welcome Thread - January 2023

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# What's wrong with the paperclips scenario?

In October 27th 2022, Eliezer Yudkowsky tweeted [this](https://twitter.com/ESYudkowsky/status/1578459400389345280):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/43304f0aac17f60adde89191f567cabcd147f1b5690b1cf6.png)

So far so good. I may ask myself when this happened and why the paperclips example was the one that actually sticked, but so far I have no reason to worry about my understanding of AI safety. But here's more of the conversation that happened on Twitter:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/592626b4082e75d7ec833630183e56848f2554dc64d4d285.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e79e963f63f34a5df7caa5fb3c9a924f82787b6b518a3143.png)

When Eliezer says "this is exactly the misunderstanding I'm complaining about" and "the related lesson is false and not valuable" I start to question my own sanity. This example is used pretty much everywhere. In Bostrom's Superintelligence, Rob Miles' videos about the stamp collector (inspired by [a blog post](https://mindingourway.com/the-stamp-collector/) by Nate Soares), Wait But Why's introduction to AI risk and I don't know what else.

So here are questions:  
  
1\. I don't see how the paperclips metaphor is wrong.  Or am I misunderstanding Eliezer?

2\. If it's wrong, why is it used everywhere?
# Feature suggestion: add a 'clarity score' to posts

Add a voting mechanism for posts, beside the karma, that indicates whether you 

*   understood the post well enough to agree or disagree with it (upvote), or
*   cannot understand the post with a reasonable time investment (downvote)

I suggest that this determines the post's clarity score (we could name it something else, like the 'simplicity' score). 

Then, allow users to sort posts by clarity.

***Especially:*** allow users to sort posts by clarity *within a topic.* That way, people can read posts in a topic in order of simplicity descending/complexity ascending. I suspect that this would make familiarization with a topic a rather linear and efficient process.
# Halifax Monthly Meetup: Moloch in the HRM

In the interest of growing the LW/EA/ACX community in Halifax, we are hosting meetups on the last Saturday of each month. Old hands and newcomers are welcome! 

We'll begin by discussing Moloch in the HRM: Our ongoing coordination problems around the housing crisis/tax allocation/etc. After that we'll shift to general, unstructured discussion.

Location: Seven Bays Cafe (2017 Gottingen Street) – [87PRMC29+9C](https://plus.codes/87PRMC29+9C)

Join us at Seven Bayes!

Contact: [conorbarnes93@gmail.com](mailto:conorbarnes93@gmail.com)
# What specific thing would you do with AI Alignment Research Assistant GPT?

Why I think this question is important: I asked myself, "What would my AGI timelines be if some AI could summarize [Yudkowsky-Ngo debates on alignment difficulty](https://www.lesswrong.com/posts/7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty) in a way that both participants agree with this summary, everyone who reads this summary understands both positions and participants can check understanding in conversation?" My semi-intuitive answer: "Five years tops and two years as modal prediction". Debate Summarizer is not a very useful Alignment Assistant, it can't boost research by 10x. If someone told me that Alignment Assistant suggested idea that sparked optimism in MIRI, I would think that we have exact amount of time it takes for someone to turn every tools needed to build such an Alignment Assistant to the creation of AGI (conditional on "this Alignment Assistant is not AGI itself").

I.e., if you bet on assistance of narrow AI in alignment research, you should also bet on finding solution quickly. Quick search for a solution requires an already existing plan. On the other hand, we are talking about a narrow AI, you can't just ask "solve alignment problem for me". You should ask specific questions, test pre-selected hypotheses, prove well-defined statements. Therefore, I think that those who want to use Alignment Assistants should outline this set of specific things as soon as possible.

UPD: Thanks janus for [the link](https://www.lesswrong.com/posts/a2io2mcxTWS4mxodF/results-from-a-survey-on-tool-use-and-workflows-in-alignment), it helped me to clarify what I would like to see as a perfect answer.

Let's suppose that your immediate answer is "brainstorming". Then the perfect specific answer is something like that: 

"In my opinion, the most narrow bottleneck in AI alignment is the lack of ideas about X, so I will brainstorm about it with Alignment Assistant."

Extremely unrealistic example:

"I have The Grand Theory of Alignment, but it critically depends on Goldbach conjecture, so I will try to prove it."

My very (very) simplified model of Paul Cristiano's answer:

"80% of alignment can be solved with ELK strategy, so we can make builder-breaker debate on (counter)examples for ELK between Assistant and ARC until we figure out the solution."

Yet another possible answer:

"I don't know! We are still in early exploratory mode, I can't imagine a specific thing. I just want to become as effectively smart as possible and see where it gets us."
# Citability of Lesswrong and the Alignment Forum

This post is short: as far as I know, there is no easy support to cite lesswrong or the alignment forum in academic papers. I think this should exist. This idea is related to the one on [putting Alignment Forum/Lesswrong posts on arxiv](https://www.alignmentforum.org/posts/TYTEJxzeK3jBMq2TZ/your-posts-should-be-on-arxiv). 

Such support already exists elsewhere, e.g. on Mathoverflow:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1107e718b865a40868f6f77f54994f8aad4d6e65c90571e8.png)

I didn't try for long to figure out if this idea was already proposed before or whether there is already support for this which I missed. 

*Thanks to Nadia Montazeri and* [*Magdalena Wache*](https://www.lesswrong.com/users/magdalena-wache) *for discussing this idea with me.*
# Latent variable prediction markets mockup + designer request

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/91dd1485e5393ab92351a76a1e7ce80ea337d558c026b3f0.png)

[I've been trying to get places like Manifold or Metaculus to add latent variables to their prediction systems.](https://www.lesswrong.com/posts/HygyWpDcwsekqmfnS/will-manifold-markets-metaculus-have-built-in-support-for) As a brief summary, latent variables would allow you to make bets and look up predictions on abstract concepts such as "How well will Ukraine do in their defense against Russia?" by linking together many concrete outcomes/indicators.

Yesterday, Nathan Young suggested that I should design the interface for it. I think that was probably a good idea, because I got the impression that I wasn't really explaining the concept properly, and also because I have a bunch of knowledge of latent variable models that might be useful to get implemented.

[Here's a link to the mockup I have so far.](https://files.onlinetests.me/lvar.html) I apologize for it being rough around the edges, and for the poor code quality; it's just something I quickly threw together.

Now for the designer request: The interface in my mockup is probably still confusing. I hope it makes it clearer what sorts of predictions I want to introduce the possibility for, especially if one plays around with the interface a bit. But ideally the meaning of all the options and graphs should be intuitive even for new users.

So if there are any designers in the audience here, I would be interested in hiring a designer to spend some time with me designing an interface that would be easier to understand.

Also if there are any programmers who are familiar with the Manifold codebase (including programmers working at Manifold), I would be interested in paying part of the development costs.

(I have a max budget of $10k for this.)

Beyond this, I would be interested in whether there is any discussion or ideas for latent variable prediction markets, including but not limited to ideas for ways to make it more intuitive to people.
# GPT learning from smarter texts?

(Epistemic status: I do not understand GPT deeply, so this is just a random idea.)

If I understand it correctly, GPT learns from existing texts. Lots of texts.

Would it be possible to make GPT smarter by simply giving it *smarter text*? Of course, writing tons of smarter text would be a lot of work, but what about annotating the existing text, like "take this more seriously" and "take this less seriously"? (From technical perspective, maybe the GPT should read the text marked as serious *five times*?) Assuming that the annonation is roughly correct, would this *improve* the results?

*

If yes, the problem is *how* to select smarter texts, especially if we want lots of them? But I think some good guesses can be made:

*   High-school textbooks. They should be educational and relatively uncontroversial ([settled science](https://www.lesswrong.com/posts/ndGYn7ZFiZyernp9f/the-beauty-of-settled-science)). There probably are some reviews of textbooks, so only annotate the ones with good reviews.
*   Some parts of Reddit are probably much better than average, in the sense that smart content gets upvoted. So include the upvoted comments.
# Wolf Incident Postmortem

Incident #210
-------------

### Status

Complete, one action item outstanding.

### Summary

Sentinel consumed by wolf after repeated false alarms.

### Impact

Loss of sentinel. No flock impact.

### Root causes

Sentinel generated noisy alerts due to premature deployment, incomplete training, and overly monotonous task. Oncalls failed to respond to true positive due to alert fatigue.

### Trigger

Wolf.

### Resolution

Gathered flock. Deployed replacement sentinel.

### Detection

Sentinel did not report at end of shift.

### Action Items

| Priority | Action Item | Type | Status |
| --- | --- | --- | --- |
| P0  | Gather flock | mitigate | complete |
| P0  | Deploy replacement sentinel | mitigate | complete |
| P1  | Update playbook for wolf alerts | prevent | complete |
| P2  | Update remaining sentinels | prevent | complete |
| P2  | Revise sentinel training program | prevent | complete |
| P2  | Investigate equipping sentinels with flutes or slings | prevent | in progress |

### Lessons Learned

#### What went well

*   Flock gathering proceeded without issues.
*   No flock injuries or losses.
*   Replacement sentinel did not exhibit false positive alerts.

#### What went wrong

*   Noisy alerts not addressed.
*   Alerts silenced contrary to playbook.
*   Loss of sentinel.

#### Where we got lucky

*   Only one wolf.
*   Wolf sated after sentinel consumption.
*   Replacement sentinel available.

### Timeline

*All times local*

March 3rd:

*   16:32 Oncalls paged "wolf".
*   16:34 First oncall arrives at sentinel location.
*   16:34 Alert diagnosed as false positive. No corrective action performed.

March 4th:

*   14:15 Oncalls paged "wolf".
*   14:19 First oncall arrives at sentinel location.
*   14:19 Alert diagnosed as false positive. No corrective action performed.

March 5th:

*   17:03 (Reconstructed) *Outage begins*, sentinel notices wolf.
*   17:03 Oncalls paged "wolf".
*   17:04 Oncalls paged "wolf".
*   17:04 Oncalls paged "real wolf".
*   17:05 (Reconstructed) Wolf consumes sentinel.
*   18:45 Sentinel does not report at end of shift.
*   19:05 Primary oncall dispatched to field.
*   19:10 Oncall diagnoses issue.
*   19:10 *Incident begins*, secondary and tertiary oncalls paged.
*   19:15 First sheep located.
*   19:52 Last sheep located.
*   20:05 Flock safe in pens.
*   20:05 *Outage ends*, flock protection fully restored.
*   20:45 Replacement sentinel identified.

March 6th:

*   07:38 Replacement sentinel deployed
*   18:45 Replacement sentinel reports at end of shift
*   18:45 *Incident ends*, 24hr without wolf alerts or activity (exit criterion).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02hPxskn4UtYdKBQFC7DsGVnm5Uq5PfUvc9caJKAkNd6fD82GM7XubSpSeSQ5KwVZ5l), [mastodon](https://mastodon.mit.edu/@jefftk/109657069519123119)*
# Big list of AI safety videos

This is a [Google doc](https://docs.google.com/document/d/1vDWLKPvJNo0k9IwMPvduO23_CXOdMpxn0aRDnvrL0yw/edit?usp=sharing) containing links to:

*   (all?) AI safety-related YouTube channels (intended to be comprehensive, at least for active channels with videos that seem worth sharing in an AI safety group)
*   some AI safety-related podcasts
*   some suggested videos, sorted into categories like "agent foundations," "individual researchers," "career planning," "rationality," etc
*   some discussion prompts for watching these videos in discussion groups (very few)
*   some resources and tips for running these discussion groups

Please comment with feedback and suggestions!

*Also posted to the* [*EA Forum*](https://forum.effectivealtruism.org/posts/vZz9tFFJKa28rTkDx/big-list-of-ai-safety-videos)*.*
# Idea: Learning How To Move Towards The Metagame

I have just finished watching someone play a variant of Brick Breaker. A variant I became intimately familiar with, indeed, obsessed, as I strove to hit the top 40 all time score which scrolled past me each time I failed, mocking me. 

And this reminded me of an interesting post I read: [To Get Good, Go After The Metagame](https://commoncog.com/to-get-good-go-after-the-metagame/). The title, however, doesn't say it all, so I'll quote the relevant section

"I’ve noticed that the ‘feel’ of improving in pursuit of a meta is strangely similar across skill trees. I noticed this after climbing high enough in the Judo skill tree to see the meta for the first time. Later, when I moved to university, learning the craft of programming became slightly easier. Ditto for content marketing, and email marketing, and so on."

This was interesting to me, as I am currently reviewing rationality techniques which generalize, and I don't think I have noticed this one mentioned elsewhere. In fact, I thought "I have never reached a meta in my life, and it sounds like so very much effort to even see the meta in anything, I guess I just have to take this on faith". But my memory is poor, and I forgot I have reached the meta a couple of times in my life by playing video games. And that when I pursue a video game to mastery, there is a clear difference in the kinds of learning that help me. I suspect, however, that this is not the skill that the post is talking about. So why mention this at all?

Because I think that you might be able to check if this ability exists pretty easily. First, find a video game with a high skill ceiling, ideally one that is new. Then, check if it has a competitive scene where people share their performances. If so, proceed. Else, find another game or consider trying to set up such a scene yourself. Then, play the game until you hit the current meta by analysing what the best players do.

This experiment seems cheap, and has some very high value. So I think I'll pursue it after my current literature review ends, gain this skill, then use it to write an addendum to whatever post I write on Ambitious Methods in Education.
# Sorting Pebbles Into Correct Heaps: The Animation

This video is an animation of [Sorting Pebbles Into Correct Heaps](https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pebbles-into-correct-heaps) by Eliezer Yudkowsky. The narration is a word-for-word reading of the story. Enjoy :)

Right after the video is made public, there will probably be some discussion about it in the Rational Animations' Discord server: [https://discord.gg/5Y3Dwz89yH](https://discord.gg/5Y3Dwz89yH)

Join in if you want to participate in the debates that will take place, but if you already understand the ins and outs of this story, make sure not to spoil it for other people! Hints and nudges to steer the conversation in productive directions are acceptable and appreciated, though!
# Running With a Backpack

In the comments on my [running by default](https://www.jefftk.com/p/running-by-default) post, [one](https://www.lesswrong.com/posts/4y3qcbnhSmXrk6p34#rHwSsvRKR7S7jxh8v) [of](https://schelling.pt/@zebrask/109637179105823464) [the](https://www.facebook.com/jefftk/posts/pfbid02RjLVh3wnXp5BsjQpMz5zYA1X5bXe6hNNh4XsCrVUEwJ9uVVZcbmSsEqdyJEVSZuCl?comment_id=566361484990223&reply_comment_id=883766642771567) [more](https://www.facebook.com/jefftk/posts/pfbid02RjLVh3wnXp5BsjQpMz5zYA1X5bXe6hNNh4XsCrVUEwJ9uVVZcbmSsEqdyJEVSZuCl?comment_id=1154096245214497) [common](https://www.facebook.com/jefftk/posts/pfbid02RjLVh3wnXp5BsjQpMz5zYA1X5bXe6hNNh4XsCrVUEwJ9uVVZcbmSsEqdyJEVSZuCl?comment_id=2087024941482505&reply_comment_id=1520389865127731) responses was that people often have backpacks and a bouncing backpack is somewhere between "annoying" and "painful, risking bruising".

There was [some discussion](https://www.lesswrong.com/posts/4y3qcbnhSmXrk6p34/running-by-default?commentId=rHwSsvRKR7S7jxh8v) on how to set up your backpack to reduce bouncing. Tips:

*   Some backpacks have straps you can buckle across your front, or you can get add-on ones. Wear them super snug.
    
*   Put your water bottle inside where it's more central and less likely to fall out.
    
*   Figure out if there are things you don't need to be transporting so you can have a lighter pack.
    
*   Use a fancy backpack with good padding and a rigid back.
    
*   Hold the shoulder straps as you run to absorb jostling.
    

These all seem like fine ideas, but I don't do any of them. My backpack is relatively low-end ([$27](https://smile.amazon.com/gp/product/B087M1H5NB/)) and non-rigid, without any sternum or waist straps. I keep my water bottle in the outside pocket, I [have many](https://www.jefftk.com/p/what-have-i-got-in-my-pocket) [little things](https://www.jefftk.com/p/portable-emergency-food) in it that I could probably carry fewer of, and I don't hold the shoulder straps. And yet I probably do the majority of my running with a backpack on: I'm commuting and I have things I need to bring with me. Even when I was trying to [run fast](https://www.jefftk.com/p/running) and [repeatedly beat](https://www.jefftk.com/p/running-and-optimizing) my personal best times I had my pack.

What works for me instead is that I've learned to run with a gait that doesn't bounce my backpack. I take a lot ([~190 bpm](https://www.lesswrong.com/posts/4y3qcbnhSmXrk6p34/running-by-default?commentId=G6fPgpStBhN4kmu45#G6fPgpStBhN4kmu45)) of short strides which means I have less time to fall between steps, and I generally stay pretty level. I think this is probably also beneficial from a perspective of minimizing the impact forces on your joints (ex: a [paper I skimmed](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8329321/)). Whether my backpack is heavy or light, it mostly stays put on my back.

It's still less pleasant to run with the extra weight of a backpack since it's more work, but if I think of it as intentional exercise (a bit like a weighted vest you already have with you!) I don't mind it too much.
# Any significant updates on long covid risk analysis?

[ETA: This question is specifically about *risk analysis of long covid, given current knowledge.* Discussion should be about things such as: 
- base rates
- priors and posteriors for LC risk
- evidence that would make one update in favor of certain new policies with regard to covid 
- other decision-relevant considerations. 

Moderation guidelines have been added to reflect this.]


Much of the attention on here to the long 'rona [LC] seems to have fizzled out about a year ago. Zvi continued to occasionally report on it, but often in the form of roundly mocking poorly designed studies or sloppy conclusions. This study, also from last year (h/t Zvi), found [50 - 80% reduction in LC in fully vaxx'd folks](https://twitter.com/bhrenton/status/1480346441201733635).

[A Q largely asking the same thing from last year.](https://www.lesswrong.com/posts/4z3FBfmEHmqnz3NEY/long-covid-risk-how-to-maintain-an-up-to-date-risk)


This recent Zvi 'rona report [cites a promising study on LC symptoms controlled against other viral infections.](https://www.lesswrong.com/posts/cxi8cTvELbMcv5rci/covid-12-8-22-another-winter-wave#Long_Covid) I do worry about Zvi having a sampling or reporting bias, I feel that a systematic review of these LC results would be a great service to the community and perhaps even humanity.

Having said that, this quote from Zvi's December is choice:


>    Long Infection is a thing across infections. Getting sick is not good for you.
>    We do not pay enough attention to Long Infection, but Long Covid is not special.

[There have also been a handful of LC doomers around these parts, eg, Siebe has a gloomy estimate of 1 - 15% chance of LC even among the vax'd.](https://www.lesswrong.com/posts/emygKGXMNgnJxq3oM/your-risk-of-developing-long-covid-is-probably-high)

I know talking about [long] covid after labor day is gauche, but it is still a thing, and naive waife that I am, I would have hoped a year later our civ's understanding of the risk profile would be "a year better." Whatever the heck that means.
# How to write a big cartesian product symbol in MathJax

$$This short post is about how to write something like $\newcommand{\bigtimes}{\mathop{\Large\times\normalsize}} \bigtimes_{i\in I} V_i$ and  $\displaystyle\bigtimes_{i\in I}V_i$

MathJax doesn't allow picture environments so we can't define a new command that draws a cross. The solution is simple: Enlarge the normal \\times command.

Paste this in a math environment at the top of the document:

\\newcommand{\\bigtimes}{\\mathop{\\Large\\times\\normalsize}}