Harley-ml/lesswrong huggingface dataset:
# Hand-writing MathML

When I write posts I use raw HTML. Yes, the modern thing to do is probably [Markdown](https://en.wikipedia.org/wiki/Markdown), but HTML was designed for hand-coding and still works well for that if you don't want anything especially fancy. But what if you want math?

Previously when I've wanted to do math I've written it out as fixed-width ASCII:

e^(-7t)

In my editor this looks like:

<pre>
e^(-7t)
</pre>

This is reasonably readable, works anywhere, and I like the aesthetic. I probably should have stuck with it, but after helping publish [a report](https://naobservatory.org/reports/predicting-virus-relative-abundance-in-wastewater/) that included some traditionally-formatted equations and learning that [MathML](https://developer.mozilla.org/en-US/docs/Web/MathML) has been supported cross-browser since the beginning of the year ([thanks Igalia!](https://mathml.igalia.com/)), I decided to try it out. I wrote the equations in two [recent](https://www.jefftk.com/p/weekly-incidence-vs-cumulative-infections) [posts](https://www.jefftk.com/p/weekly-incidence-including-delay) in it, and am mixed on the experience.

It definitely does look nicer:

e - 7 t

On the other hand, here's how it looks in my editor:

<math display=block>
<msup>
  <mi>e</mi>
  <mrow>
    <mo>-</mo>
    <mn>7</mn>
    <mi>t</mi>
  </mrow>
</msup>
</math>

There's a small learning curve on when to use the different tags, but mostly it's just very verbose. And I think, needlessly so? That "`-`" is an operator, "`7`" is a number, and "`t`" is an identifier could all be the default. Then I could just write:

<math display=block>
<msup>
  e
  <mrow>
    -7t
  </mrow>
</msup>
</math>

And we could remove many uses of `<mrow>` too: a series of characters without whitespace separating them could be already treated as a group:

<math display=block>
<msup>
  e
  -7t
</msup>
</math>

Of course if you wanted to use a character for a non-traditional purpose you could still mark it up as one, but a good set of defaults would make MathML much more pleasant. I'd hate to have to read and write blog posts as:

  <word><lt>h</lt><lt>e</lt><lt>l</lt><lt>l</lt><lt>o</lt></word>
  <word><lt>w</lt><lt>o</lt><lt>r</lt><lt>l</lt><lt>d</lt></word>
  <pnct>.</pnct>

I know I'm about [25 years too late](https://web.archive.org/web/19990428145738/http://www.w3.org/Math/mathml-faq.html) on this, and I'm happy that a pure-HTML solution is now cross-browser, but it's still sad we ended up so close to a comfortable hand-editable solution.

(Just use [MathJax](https://www.mathjax.org/)? Nope—I don't want a runtime dependency on JS. Though I could see including a LaTeX-to-MathML or a MathML-verbosifier step at build time.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02EBNF19Y2Z7bf1EzSes2eiL3fdX8RFXBXWUpCzb1Koj4VcFRyCCoxms3KFfWirDwMl), [mastodon](https://mastodon.mit.edu/@jefftk/111114175926556334)*
# Places to meet interesting middle-aged men?

The rationalist community is a nice place to meet interesting, intellectual men, which makes it potentially a good dating pool for straight/bisexual women. However, this community is young: most people seem to be in their 20s or 30s. For middle-aged women like me, that's a problem. What are some other communities where it's easy to meet straight/bisexual men who are: (i) intellectually interesting is vaguely similar ways to how the rationalist community is intellectually interesting (ii) middle-aged and (iii) available?
# Honor System for Vaccination?

In June the social dance I help organize decided to [drop its requirement](https://blog.bidadance.org/2023/06/some-mask-optional-dances.html) that people be vaccinated and boosted:

> Checking vaccination cards at the door requires an additional volunteer, is a hassle for attendees, and at this point the communities we draw from are overwhelmingly vaccinated. While we still recommend staying up to date on vaccinations (including your flu shot and covid booster in the fall!), we're dropping this requirement starting with our July 2nd dance.

One question we got, both in person and electronically, was why not keep requiring vaccination on the honor system? Speaking for myself and not BIDA, I think this is rarely a good policy for events.

(I'm setting aside the question of whether vaccination is helpful in reducing risk to others, and writing all of this under the assumption that it is. I do think this is unclear, especially if a policy does not require a recent booster.)

The issue is, an honor system is a bad fit when getting caught is very unlikely and the harms of violating the rule are diffuse and unclear. If an unvaccinated dancer ignores the policy and does happen to bring an infection to the dance that vaccination would have prevented, it's very unlikely that anyone will ever know. There's no feedback mechanism promoting honesty here. Instead, you're filtering unvaccinated people for ones who are ok violating the policy, and nudging more unvaccinated people toward being dishonest in this way.

Compare this to a dance in a small community that does payment on the honor system: an unattended fiddle case by the door for you to drop your money into. If someone doesn't contribute it's much more likely that people will notice, and it's also much clearer that when you defect you're hurting people. There's still some pressure toward dishonesty, but there's enough pushing the other way that it still works well.

The honor system can work well in some cases, but I don't think this is one of them.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0gwa7i884k2nmFPbrBcJbzqu1xFQ2FN8rRfTaAkDmCJwJNdrqsSduNjUC3QykbDvtl), [mastodon](https://mastodon.mit.edu/@jefftk/111119946932019103)*
# How have you become more hard-working?

I'd be curious to hear stories of people who have successfully become more hard-working, especially if they started out as not particularly hard-working. Types of things I can imagine playing a role or know have played a role for some people:

*   Switching roles to something that is conducive to hard work, e.g. a fast-paced environment with lots of concrete tasks and fires to put out.
*   Medication, e.g. ADHD medication
*   Internal work, e.g. specific types of therapy, meditation, self-help reading, or other types of reflection.
*   Productivity hacks, e.g. more accountability, putting specific systems in place
*   Motivational events, arguments, or life periods, e.g. working a normal corporate jobs where long hours are expected
*   Switching work environment to something that is conducive to hard work, e.g. always working in an office with others who hold you accountable

This curiosity was triggered by realising that I know of very few people that have become substantially harder-working over their late adolescence/adult life. I also noticed that the few people that I know successfully and seemingly permanently increased their mental health/work satisfaction always were hard-working even when they were unhappy (unless they were in the middle of burn-out or similar).

People becoming more hard-working seems really useful but I haven't seen much in terms of evidence that it's feasible or effective methods. If there are books or studies on this topic, those would also be welcome. Thank you!
# Feedly Breaks MathML

A few days ago I wrote about [my experience with MathML](https://www.jefftk.com/p/hand-writing-mathml), and despite being somewhat positive on it in that post I've decided to stop using it for now. The problem is, it doesn't display for people who follow my blog through RSS on (I'm guessing) most popular RSS system.

Here's a screenshot from my most recent MathML-containing post, on my website:

[![](https://www.jefftk.com/raw-html-mathml-working.png)](https://www.jefftk.com/raw-html-mathml-working-big.png)

And here's the same portion of that post running in the web version of Feedly on the same browser:

[![](https://www.jefftk.com/feedly-missing-mathml.png)](https://www.jefftk.com/feedly-missing-mathml-big.png)

Poking at developer tools, here's what Feedly is sending over the network to my browser:

<p> It definitely does look nicer: </p> <p> </p> <p> On the other hand

This shows that they're removing the MathML on the server, instead of there being some issue once it gets to the client.

This also explains why it doesn't work in other RSS readers that use Feedly as a backend. For example, I see the same thing in Palabre.

I tried to file a bug with them, but as far as I can tell you need to be a paid subscriber to do that. Authors trying to let Feedly know about bugs that impact their readers seems not to be a thing they're open to?

Here's a test case to reproduce this issue: [mathml-simple.rss](https://www.jefftk.com/mathml-simple.rss).

For now I'm going to stop using MathML, and go back to ascii math.

This makes me sad: during Nora's nap yesterday I wrote the [MathML Verbosifier](https://github.com/jeffkaufman/mathml_verbosifier) I'd been thinking of, which lets me write:

    a^2 + b^2 = c^2

as the relatively readable:

    <msup>a 2</msup> + <msup>b 2</msup> = <msup>c 2</msup>

and automatically convert that into the spec-compliant:

    <msup><mi>a</mi><mn>2</mn></msup><mo>+</mo><msup><mi>b</mi><mn>2</mn></msup><mo>=</mo><msup><mi>c</mi><mn>2</mn></msup>

(In the LW comments there was also [discussion](https://www.lesswrong.com/posts/aQDNfCrhqYg95sMDN/hand-writing-mathml?commentId=xhAeWP4tPTp8HHWg2) of whether I should avoid using MathML for the benefit of people running pre-v109 Chrome. I don't think that makes sense: there have been several serious vulnerabilities since then, including the [WebP zero-day](https://blog.isosceles.com/the-webp-0day/), and I'm not interested in making changes to my site to support people running dangerous configurations.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0CTfGH3HFuLV3kfcMiLrHghYTuff5KXpaTdXQ5skcuDBFM7aBtPYkvYuc6wM6UmWWl), [mastodon](https://mastodon.mit.edu/@jefftk/111126039615608272)*
# Amazon to invest up to $4 billion in Anthropic

> Today, we’re announcing that Amazon will invest up to $4 billion in Anthropic. The agreement is part of a broader collaboration to develop reliable and high-performing foundation models.

  
(Thread continues from there with more details -- seems like a notable major development!)
# You should just smile at strangers a  lot

This is so basic I feel stupid for saying it in such an intellectual place. But most of you guys share my enthusiasm for effective altruism, and here's something really altruistic and really effective that I've been doing for two decades and I think lots of you should do too.

Just smile at strangers. Walk down the street with a default smile on your face, seek eye contact and just keep smiling. You don't need to do anything else, not talk to them or anything.

This is tiny, but it's a pretty unadulterated good thing. Facial feedback research is clear, smiling does make you feel better. It costs basically nothing, and definitely less than you get back in good feelings for yourself. It's a quality of life improvement for everyone involved.

After what has to be hundreds of thousands of such micro-interactions, I can report that nearly half of the strangers will smile back, although often they'll look away as they do. This is probably highly dependent on the culture where you do this, especially on whether it is a high-trust environment.

I have learned not to smile at beggars, though. They usually took it as an invitation to target and haggle me.

To be fair, I'm a male and I don't know if a female doing this would get very different reactions. Maybe some of you want to try it and report back?
# What is wrong with this "utility switch button problem" approach?

Suppose you have some positive utility functions $A,B$ , mathematically considered to be random variables dependant on choice of policy. 

Let $K$ be a random variable that is 1 if the button is pressed, 0 otherwise. Again dependant on choice of policy. I am considering that there is a particular time that the button must be pressed to make this a single bit. You have a single chance to switch this AI between 2 utilities, and whatever you pick in that instant is baked in forevermore.

Let $a=\mathbb E(A(1-K)), ~b=\mathbb E(BK)$ be the expected partial utilities.

Now consider all policies. In particular, consider the pareto frontier between $a$ and $b$. Through the introduction of randomness, we can make that parito frontier continuous.

We want some policy off that frontier. 

There will be some policy $h_a$ that throws all resources at maximizing $a$. And there will be some (usually different) policy $h_b$ that throws all available resources at maximizing $b$. 

let $q$ be a tradeoff rate. And $u=a(1-q)+bq$. The parito frontier can now be defined as the policies that maximize $u$ for varying $q$. 

For $h_a$, this is the policy that is optimal when $q=0$ which has $u(h_a)=a(h_a)$. Then $\frac{du}{d\mathbb E(K)}<0$. 

Likewise at $h_b$ with $q=1$ have $\frac{du}{d\mathbb E(K)}>0$.

So somewhere in the middle, by the continuity granted by stochastic mixes of policies, their must be at least one point where $\frac{du}{d\mathbb E(K)}=0$. Use that policy, or stochastic mixture of policies. 

This agent will never pay $\epsilon$ utility to set the value of the button one way or the other. Because the policy is one in which has $\mathbb E(u|K=0)=\mathbb E(u|K=1)$ (I think) Which hopefully means that this policy does worse than the policy that doesn't pay to change the button, but otherwise does everything else the same. 

Put another way, if you pay to flip the button. Then you must care about the state of the button, if the chance the button is pressed changes, you are getting more or less for your $\epsilon$ utility. So the gradient can't be 0. 

And this is described as a choice of policy. A framework which automatically includes the actions of all sub agents. 

So, how does this fail? Who can break this agent and get it to do crazy things?
# Mechanistic Interpretability Reading group

Greetings.

This is an open invitation for folks interested in technical AI alignment literature to join our reading group, which has been running for 22 weeks now. Our meetings take place online via the *Mechanistic Interpretability Group* Discord server, accessible through this [invite link](https://discord.gg/rUv2Jcma4Y).

Each week 4-5 papers are shortlisted and polled, occasionally with some central theme, and people vote on which papers they'd like to read and discuss. The most popular option is selected for the next week's reading. You can find a list of the papers we've already read as well as some of the upcoming papers at [https://mechinterp.com/reading-group.](https://mechinterp.com/reading-group) 

In addition to paper discussions, we occasionally host authors for Q&A sessions. This week, we will be reading *Studying Large Language Model Generalization with Influence Functions* ([arxiv](https://arxiv.org/abs/2308.03296)). One of the authors, [Juhan Bae](https://www.juhanbae.com/), will be joining us for a Q&A session on September 27th at 5pm GMT.

The usual meeting time is Wednesdays at 5pm GMT (10am PT) and the sessions usually run for an hour. If you're unavailable at that time, we've recently started recording some of our sessions, transcribing them with Whisper. The recordings and transcripts are available [here](https://drive.google.com/drive/folders/1v5nZ0yUO5c3rIW7uYfk7mXZKCQibYwwi?usp=sharing).

We welcome anyone interested to join us in our discussions. Feel free to reach out if you have any questions or need further information. The best way to reach us is through the Discord server.
# What's your standard for good work performance?

A lot of people in the community, including me, are working independently (or have a lot of autonomy, even if they are employed). A lot of people, including me, often feel like they are underperforming or at least wonder if they are. But how do I actually know when I'm *not* underperforming? I'd like to make some criteria for under which circumstances I'll consider my work satisfactory. I'd be curious to hear what other people would consider "decent" output.

This is obviously hard to define but some types of data that seem helpful:

*   How much do you get done in a typical month/half year?
*   How much do consider aspirational but realistic to get done in a typical month/half year?
*   How much do you consider on the low end but okay to get done in a typical month/half year?
*   What kind of output would you want to see out of a researcher/community organiser/other independent worker within a month/half a year to be impressed/not be disappointed? (Assuming this is amount is representative of them)
*   What's the minimum output would you want to see out of a researcher/community organiser/other independent worker to be in favour of them getting funding to continue their work?  (Assuming this is amount is representative of them)
*   What's the minimum output would you want to see out of your friend to feel good about them continuing their current work? (Assuming this is amount is representative of them)

I literally mean silly things like "X number of substantive research documents of roughly this and that quality" - totally okay if very domain-specific. I know this probably hugely varies from person to person and is hard to answer in general - There's lots to take into account that seems hard to specify like quality of work and often we really care about outcomes instead of outputs. Feel free to rephrase the question in whichever way seems useful to you.

(I know that there is no objective "enough" or that the real measuring stick are my values and reality. (I would have liked to include a link to a post that I'm sure exists but couldn't find it.) And there's a bunch of things that are individual to me and my line of work of course, so I don't expect people to say things that are directly useful and applicable to my situation. But I still think that sometimes a bit of data would be really useful to me and hopefully many others!)
# The Hidden Complexity of Wishes - The Animation

This video introduces a series focused on the outer alignment problem. In future videos, we'll explore how this problem affects machine learning systems today and how it could lead to catastrophic outcomes for humanity.

The video is an adaptation of [@Eliezer Yudkowsky](https://www.lesswrong.com/users/eliezer_yudkowsky?mention=user)'s post [The Hidden Complexity of Wishes](https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes). The text has been slightly changed, with Eliezer's approval, mainly because the original has many references to other articles.
# Announcing FAR Labs, an AI safety coworking space

FAR Labs is a coworking hub in downtown Berkeley for organizations and individuals working on AI safety and related issues. Since opening the space in March 2023, we have grown to host approximately 30 members. Our members are primarily drawn from four anchor organizations, but we also host a number of independent researchers and research teams.

Now that our initial setup is complete, we are pleased to announce an open [call for applications](https://far.ai/labs/) from individuals or organizations.

Our initial aims for FAR Labs:

*   First and foremost it should be a place to do great work. Our members are working on challenging problems, and we want to improve their effectiveness, reduce distractions, and provide a professional environment for them to work in. That includes providing a variety of workspaces (private offices, dedicated desks and hot-desks, general areas), catering, and other office amenities such as a gym.
*   A warm, intellectually generative culture. Having interesting and fun conversations is one of the best parts of working in a shared environment, and championing a culture that enables those interactions is incredibly important to us.
*   Supporting collaborations between members, other alignment organizations, and outside collaborators (e.g. academics, or industry researchers). While membership is tied to actively working on AI safety (technical or governance) or related areas (e.g. field building, advocacy, fundraising), we also want to make a space that’s welcoming to many viewpoints, which we expect to benefit both members and visitors.

[FAR AI](https://far.ai/)’s broader mission is to support research and initiatives that promote trustworthy and safe AI systems. FAR Labs is an investment in operations and coordination. By creating research environments and good operational scaffolding, we can accelerate safety research and x-risk reduction across projects and orgs.

For the past six months that’s looked like setting up the space and getting the basics in place (office, food, equipment). Moving into 2024 the Labs team will begin offering programs for members – as well as others in the AI safety ecosystem – for developing relevant skills for research and operational excellence. We’re particularly excited about identifying best practices and providing training to help members in building and scaling high performing teams.

FAR Labs runs at cost/a slight loss[^giybb5c1f8g]; we’re aiming for a fully member supported office and community space.

We are opening for new membership applications. Currently we hope to onboard one to three alignment oriented organizations, and perhaps a handful of independent members[^yeh6oy4xekr], aiming for a total membership of 40-50 people. If you’re interested in working from FAR Labs, or would like to learn more, [please reach out](https://far.ai/labs/).

[^giybb5c1f8g]:  Programs, external visitors, and workshops will be grant funded, while our ongoing day to day office costs are covered by member dues. 

[^yeh6oy4xekr]: While we host several independent researchers we do prioritize organizations.
# Paul Christiano's views on "doom" (video explainer)

I have just published a [video explainer](https://youtu.be/K8SUBNPAJnE) of Paul Christiano's Lesswrong post: [My views on "doom".](https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom)

This is the second Lesswrong post I make a video explainer about, [the first one](https://youtu.be/JXYcLQItZsk) being Richard Ngo's [Clarifying and Predicting AGI.](https://www.lesswrong.com/posts/BoA3agdkAzL6HQtQP/clarifying-and-predicting-agi) 

Feel free to mention in the comments if you have any feedback on this kind of content or would like to suggest another (AI related) Lesswrong post I could make a video about.
# My submission to the ALTER Prize

This is a linkpost for my submission to the ALTER AI Alignment Prize. As such, it's 8 and a half pages of pretty unapologetically dense math and/or philosophy the whole way through. It represents my own work, and I stand by the results and thinking in it. As such, I welcome actionable feedback, including checking of my reasoning and typography. I'm also happy to answer questions in the comments here or on Discord.

Feedback: https://www.admonymous.co/lorxus
# Fifty Flips

*An unfair coin (potentially EXTREMELY unfair) will be flipped fifty times. Your goal is to correctly predict as many of these flips as possible, by deducing the nature of the unfairness as quickly as possible.*

*\[Predict Heads\] \[Predict Tails\]*

* * *

You can play this (in-browser, very short) game [here](https://h-b-p.github.io/FiftyFlips/); the rule governing the unfairness is automatically revealed after flip 50. Followups with different governing rules are [here](https://h-b-p.github.io/FiftyFlips/indexB), [here](https://h-b-p.github.io/FiftyFlips/indexC), [here](https://h-b-p.github.io/FiftyFlips/indexD), [here](https://h-b-p.github.io/FiftyFlips/indexE) and [here](https://h-b-p.github.io/FiftyFlips/indexF).
# The 99% principle for personal problems

Often when people are dealing with an issue – emotional, mental, or physical – there’s genuine progress and the issue becomes less and seems to go away.

Until it comes back, seemingly as bad as before.

Maybe the person developed a coping mechanism that worked, but only under specific circumstances. Maybe the person managed to eliminate one of the triggers for the thing, but it turned out that there were other triggers. Maybe the progress was contingent on them feeling better in some other way, and something as seemingly trivial as sleeping worse brought it back.

I’ve been there, many times. It is often very, very frustrating. I might feel like all the progress was just me somehow perpetuating an elaborate fraud on myself, and like all efforts to change the thing are hopeless and it will never go away.

And I know that a lot of other people feel this way, too.

Something that I tell my clients who are experiencing this despair is something that I got from [Tucker Peck](https://meditatewithtucker.com/), that I call the 99% principle:

The most important step is not when you go from having the issue 1% of the time to 0% of the time, but when you go from having the issue 100% of the time to 99% of the time.

It’s when you go from losing your temper or going into a fawn reaction in every disagreement, to staying cool on some rare occasions.

It’s when you go from always procrastinating on an unpleasant task, to sometimes tackling it head-on.

It’s when you go from always feeling overwhelmed by anxiety to having some moments where you can breathe and feel a bit more at ease.

When you manage to reduce the frequency or the severity of the issue even just a little, that’s the beginning of the point where you can make it progressively less. From that point on, it’s just a matter of more time and work.

Of course, not all issues are ones that can ever be gotten down to happening 0% of the time, or even 50% of the time. Or even if they can, it’s not a given that the same approach that got you to 99%, will get you all the way to 0%.

But even if you only get it down somewhat. That somewhat is still _progress_. It’s still a genuine improvement to your life. The fact that the issue keeps occurring, doesn’t mean that your gains would be fake in any way.

And also, many issues _can_ be gotten down to 0%, or close to it. Over time both the frequency and severity are likely to decrease, even if that might be hard to remember in the moments when the thing gets triggered again.

For many issues, it can be the case that the moment when it finally goes to 0% is something that you won’t even notice – because the thing had already become so rare before, that you managed to forget that you ever even had the problem.
# Expectations for Gemini: hopefully not a big deal

**Introduction**
----------------

My goal is to register and share my expectations and hear others' opinions on their expectation for the relative performances of Gemini VS GPT-4.

**My expectations**
-------------------

GPT-4 to Gemini will likely not be as big a jump in capabilities as GPT-3 to GPT-4 was. 

Gemini could bring surprises by being more agentic than GPT-4. Being better at planning and longer horizon tasks. But this is likely difficult to achieve, or strong LLM agents would already be making the buzz.

**Comparison**
--------------

### **From GPT-3 to GPT-4**

*   **Scaling Factor**: x100 more compute than GPT-3.
*   **Optimization**: Chinchilla scaling laws (for MoE) over OpenAI/Kaplan scaling laws.
*   **MoE Over Dense**: Utilizes Mixture of Experts (MoE) instead of dense layers.
*   **Data Quality**: Likely higher-quality data, not sure.
*   **Image Generation**: Not publicly released, possibly due to subpar performance or security risks.
*   Tools are added during finetuning.
*   **Algorithmic Gains**: 3 years between GPT-3 and GPT-4.
*   GPT-4 may already employ process-based feedback.
*   GPT-4 aimed for training compute efficiency. GPT-4 was not designed to be commercially deployed at scale.

### **GPT-4 to Gemini**

*   **Scaling Factor**: ~x5 (x20) more compute than GPT-4.
*   **Supercomputer Constraint**: No existing supercomputer could feasibly provide x100 more compute than used for GPT-4. (Not sure but likely)
*   **Multimodal**: maybe image, audio, speech.
*   **Data Efficiency**: Possibly better quality data like Google Books, fewer epochs.
*   Tools could be added either during finetuning or pretraining.
*   **Algorithmic Gains**: ~1 year between GPT-4 and Gemini.
*   Gemini more likely aims for inference efficiency, given its intended extensive usage by Google. Maybe sacrificing training efficiency.
*   Gemini trained to be more agentic, better at planning, etc. ("GPT-4 + AlphaGo").

  
  
**Note:** I drafted that before news of Gemini's release and capabilities but failed to finish writing... Since then, there have been some reports of Gemini being roughly at the level of GPT-4...
# Dall-E 3

It seems like we do not yet have a post about Dall-E 3. It was announced a few days ago and can now be tried via bing image creator (long wait times). 

I think it is worth mentioning, because it seems like a big jump. 

I spend quite a lot of time playing around with Dalle mini, Dall-E 2, Stable Diffusion in its many flavours and it was generally the case that it was very hard to get specifically what I wanted, to the point where I was disillusioned and few of my projects materialised. 

A teddy bear and a panda bear holding hands - it took until SDXL for that to work out at least occasionally (the wedding was long over).

Maritime paintings of awesome sailing ships - the rigging was generally completely messed up (finetuning on Montague Dawson and Co didn't help).

Comic books aka Prince Valiant fan-fic - impossible to reliably get specific poses or situations involving several characters. 

Mammoths in space - don't ask, but I had to do the space hangar in SD and the mammoths in Dall-E and bash it all together. 

The Stable Diffusion eco system build around these shortcoming with control nets and finetuned models and increasingly complicated workflows - but now it seems with Dall-E 3 many more things just work right out of the box. 

From my few experiments and the examples I have seen elsewhere this might be the point where image generation just works.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2f49c994a6a33ec0621bbab1f57015a8fd5511bf3cbd7937.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/62bd9e027825f3d640c0d082c67dbefe010266237356cf3c.jpg)
# Potential alignment targets for a sovereign superintelligent AI

I'd like to compile a list of potential alignment targets for a [sovereign](https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintelligence-15-oracles-genies-and-sovereigns) superintelligent AI.

By an alignment target, I mean something like what goals/values/utility function we might want to instill in a sovereign superintelligent AI (assuming we've solved the alignment problem).

Here are some alignment targets I've come across:

*   [Alignment to a human user](https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6) (or group of human users).
*   [Ambitious value learning](https://www.lesswrong.com/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning).
*   [Coherent extrapolated volition](https://www.lesswrong.com/posts/EQFfj5eC5mqBMxF2s/superintelligence-23-coherent-extrapolated-volition).

Examples, reviews, critiques, and comparisons of alignment targets are welcome.
# [Link] Bay Area Winter Solstice 2023

Bay Area Winter Solstice 2023 will be held on December 16 at the Freight & Salvage in downtown Berkeley!

Tickets are available at [https://www.eventbrite.com/e/bay-area-winter-solstice-2023-tickets-721678057497.](https://www.eventbrite.com/e/bay-area-winter-solstice-2023-tickets-721678057497.)

See the official event page at [https://fb.me/1QOVAoRFRHROxgK](https://fb.me/1QOVAoRFRHROxgK) for more information and future updates.
# What evidence is there of LLM's containing world models?

Since this still seems to be an area of active debate within the ML community, it may be worthwhile providing a location to gather this evidence all in one place. Please only list one paper per answer as that makes it easier for people to comment on it (and possibly critique the evidence). Also feel free to include evidence of them *not* containing world models.
# Can the House Legislate?

The US government is in a less functional position than usual:

*   Legislation requires (among other things) a plurality vote of the House of Representatives.
    
*   This requires the Speaker of the House to call a vote.
    
*   This requires a Speaker of the House.
    
*   The House removed its Speaker [on Tuesday](https://apnews.com/article/mccarthy-gaetz-speaker-motion-to-vacate-congress-327e294a39f8de079ef5e4abfb1fa555).
    

We've never done this before, so it's unclear how it will go. The House does have a "Speaker pro-tempore" backup, added [in 2003](https://www.nbcnews.com/politics/congress/kevin-mccarthy-ousted-speaker-house-next-steps-rcna118561), but it's not clear whether they are allowed to use this position to call votes other than for a new speaker, or, if they would even if allowed:

> In the meantime, other action on the House floor, including any legislative business, is likely to be halted. House staff aides believe the acting speaker may wield the gavel only to administer the election of a new speaker.
> 
> But others argue that as acting speaker, Mr. McHenry can exercise powers beyond overseeing a speaker's race, as long as a majority tolerates it.
> 
> "From an institutional House rules perspective, Speaker McHenry has the powers of the speakership, and he will continue to exercise those powers to the extent and degree that the majority party is willing to tolerate," said Josh C. Huder, a senior fellow at the Government Affairs Institute at Georgetown University. "If he does something too brash or too bold, they will rein him in. And that's really the only thing that's governing his authority."
> 
> House rules do not explicitly prohibit the interim speaker from adopting the powers of an elected speaker. But the rule that led to Mr. McHenry's ascent was developed with only temporary absences in mind, not a vacant chair, according to Stan M. Brand, the former general counsel to the House. That leaves legal room for Mr. McHenry to exercise broader powers, such as conducting legislative business, if he chooses.—[NYT, 2023-10-04](https://www.nytimes.com/2023/10/04/us/politics/patrick-mchenry-interim-speaker.html)

I've started a [prediction market](https://manifold.markets/JeffKaufman/will-mchenry-as-speaker-protempore), which as of this writing is at 20% with four traders:

I set the resolution conditions as:

> Resolves true if McHenry uses his current pro-tempore position to allow the House to schedule a vote for anything other than a new Speaker. Resolves false otherwise, including if a new Speaker is elected quickly enough that we don't find out whether legislating would have been acceptable.

I interpret this as a combination of whether it is strictly permitted and how long it might take to elect a new Speaker: if one can be elected quickly ([market](https://manifold.markets/zzlk/how-long-will-the-house-be-without)) then there's much need for McHenry to call votes. I think it's probably legally allowed but socially not (yet) allowed, and the latter is more likely to change the longer we go without being able to elect a Speaker.
# Linking Alt Accounts

Sometimes people have something they want to say without it being traceable back to their main identity, and on internet forums it's common for people to use multiple accounts ("alts") for this. As machine learning software gets better, however, it becomes increasingly practical to link a person's accounts.

A few months ago someone ran a [simple](https://archive.ph/ml2bI) [stylometry](https://en.wikipedia.org/wiki/Stylometry) tool [across Hacker News comments](https://news.ycombinator.com/item?id=33755016), and identified many alts, including one for the site's founder. To further demonstrate that this isn't just an academic consideration, I recently did the same for LessWrong and the EA Forum. I'm not going to share the code or the probabilities it generated, and I've only looked at the output enough to be pretty sure it's working. Trained on half of the data and tested on the other half it was consistently able to link accounts, however, and it was also able to identify at least one non-public alt account I already knew about.

This is an example of a general problem with privacy: even if something seems careful enough now, you [can't trust the future](https://www.jefftk.com/p/preparing-for-less-privacy) to keep things private.

(If you do want somewhat more protection now, however, I think best practice is running your alt comments through an LLM to change the style.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid032HiExJzuaiphBmrZbEpzMkhUcPcm8d3QYJGA67Dh74vj1cB4oaxd2kHArwHYNNnxl), [mastodon](https://mastodon.mit.edu/@jefftk/111189117573472445)*
# Related Discussion from Thomas Kwa's MIRI Research Experience

One of the subthreads in [Thomas Kwa's MIRI research experience](https://www.lesswrong.com/posts/qbcuk8WwFnTZcXTd6/thomas-kwa-s-miri-research-experience) was about his experience working with Nate. In the comments, some other people brought up their own negative experiences. There was a lot of ensuing discussion about it.

Thomas felt this was [distracting from the points he was most interested in](https://www.lesswrong.com/posts/qbcuk8WwFnTZcXTd6/thomas-kwa-s-miri-research-experience?commentId=Ac4pcdmEDts38iWpM) (e.g. how infohazard policies slow down research, how new researchers can stop flailing around so much, whether deconfusion is a bottleneck to alignment, or the sharp left turn, etc). 

I also somewhat regretted curating the post since we normally avoid curating "community politics" posts, and while the post had a lot of timeless content in both the OP and the discussion, it ended up being a major focus of the comments.

So, I'm moving those comments to this escape-valve-post, where the discussion can continue in whatever direction people end up taking while leaving the original post to focus on more timeless topics that are relevant whether or not you're plugged into particular social scenes.
# Contra Nora Belrose on Orthogonality Thesis Being Trivial

[Disagreeing with Eliezer Y., Nora Belrose calls the Orthogonality Thesis "trivial, false or meaningless/confused"](https://twitter.com/norabelrose/status/1709598947470377139).

I think this is wrong. For instance, without the orthogonality thesis, one might think that in order to improve the world, one should charge ahead in creating an AGI, which would automatically use its superior intelligence to figure out the meaning of life and achieve it.

You might think this is stupid, but some significant people believe it. For example, some guy named E. Yudkowsky who used to run an organization for creating AGI [wrote an argument like this](https://docs.google.com/document/d/12xxhvL34i7AcjXtJ9phwelZ7IzHZ_xiz-8lGwpWxucI/edit?hl=en_US#heading=h.6fc6c4d29c3f), but thanks to Eliezer Y.'s reasoning about the orthogonality thesis and related topics, he eventually changed his mind. Clearly the orthogonality thesis is nontrivially useful for cases like this.
# Griffin Island

The park near our house has an interesting sign:

[![](https://www.jefftk.com/griffin-island-sign.jpg)](https://www.jefftk.com/griffin-island-sign-big.jpg)

A parent put it up, labeling the area you're able to explore from this point without crossing any streets. It's the large red area on my [child walksheds](https://www.jefftk.com/p/child-walksheds) map.

The name has caught on well, and it's very useful to be able to refer to this block!

Yesterday I added it to [OpenStreetMap](https://www.openstreetmap.org/way/1213743402) and [Google Maps](https://maps.app.goo.gl/dKGNA7xHQuEdjqyL7) and I now see it in both.
# A thought about the constraints of debtlessness in online communities

I notice that there's a lot of work done in online communities that's hard to price. We know so little about its price that it's a bit ambiguous as to whether it should be paid for or charged for:

*   Is writing posts a duty or a privilege? You get to put anything you want in them, but not really, because it has to be something people want to read, and you're competing with so many other options so it's tough.
*   Is curating articles a duty or a privilege? If you're a prominent curator, writers will essentially be writing for your tastes. On the other hand, you're going to have to read a lot of not-yet-curated articles. And again, you have to operate within the bounds of what others want and competition with other curators is intense.
*   Is consuming curated articles a duty or a privilege? It's definitely *supposed* to be a gift. But you find yourself reading a lot of news that stresses you out, as an intermediate goal towards later writing your own takes about it and competing in the producer arena.
*   Is moderating a duty or a privilege? On the one hand, you have the highest form of power over your community. On the other, the community is mad at you, and can leave, and mostly can't really be herded.

Reflecting on this, I think it might not be a coincidence that all of the most common roles in online communities have equal or greater benefits than costs: We don't have ways of compensating an imbalance. There's no process for tracking and repaying debt (*and even with better technologies, crediting the production of information — a non-excludable good — will remain difficult*), which explains why it would be the case that:

It seems as if little is done, in an online community, unless it's from the narrow category of acts that are in themselves self-compensating.

Which is concerning, because most conceivable beneficial exchanges pair acts which are not in that category.
# Bird-eye view visualization of LLM activations

I’m starting to learn about [mechanistic interpretability](https://www.neelnanda.io/mechanistic-interpretability/quickstart), and I’m seeing lots of great visualizations of transformer internals, but somehow I’ve never seen the whole large model’s internal state shown at once, on one image.

So I made this visualization, for Llama-2-7B. Attention matrices are on the left, in 32 rows for 32 blocks, top to bottom. To the right, there are 64 rows: residual stream (odd rows) and internal MLP activations (even rows). Finally, output MLP and unembedding layer are on the bottom.

Activation maps are downscaled horizontally, with maxpooling, to fit into 1000px wide image.

Example for a prompt “2+2=”:

![](https://grgv.xyz/assets/llama_vis/llama_vis_2_2p2_notes.png)

And an example for the prompt: "William Shakespeare was born in the year”:

![](https://grgv.xyz/assets/llama_vis/llama_vis_2_sh.png)

And for the prompt "blue pencils fly over moonlit toasters”:

![](https://grgv.xyz/assets/llama_vis/llama_vis_2_nonsense.png)

Probably not especially useful for interpretability, but at least it looks pretty )
# NYT on the Manifest forecasting conference

Kevin Roose from New York Times just published an article about our [Manifest](https://manifestconference.net) conference, forecasting and prediction markets. And despite what you might assume from "NYT covering weird rationalist-y event", the article is very fair, perhaps even positive! It is paywalled; here's an [archive.is link](https://archive.ph/sf5lw), and excerpt below:

* * *

I had never before attended a business conference with a 28 percent chance of an orgy.

But those were the [official orgy odds](https://archive.ph/o/sf5lw/https://manifold.markets/Stralor/will-anyone-have-an-orgy-at-manifes) when I arrived at Manifest, a self-described “gathering of forecasting nerds” that the forecasting start-up Manifold Markets put on last month in Berkeley, Calif.

By the second day of the conference, the odds had risen to 47 percent. And on the third day, they reached 100 percent — because there had, in fact, been an orgy. (No, I was not invited.)

This strange blend of data and debauchery — equal parts Math Olympiad and Burning Man — was the dominant vibe at Manifest, which was held in a converted hotel and populated by a crowd of about 250 tech workers, bloggers, economists, students and assorted wonks.

They were there to celebrate prediction markets, online platforms where users can wager on future events — everything from “Will Ukraine regain control over Crimea before the end of 2024?” to “Will Elon Musk and Mark Zuckerberg have a physical fight in 2023?”

On Manifold Markets, users can create a market on any topic and invite other users to bet on it. Winners get bragging rights along with units of Mana, the company’s play-money currency, which they can convert to charity donations or use on other bets.

Prediction markets aren’t a new idea, nor is the hope that betting could produce useful information. Gambling on elections and other political events was common in the United States during the 19th and early 20th centuries. And in countries where political betting is still legal, odds are often cited alongside polls and surveys as a meaningful data point.

But in recent years, prediction markets have caught the attention of a crowd of Silicon Valley empiricists who believe we can fix much of what ails society by betting on our future the way we wager on stocks or sports games.

These people believe the world is full of bad information — biased news, out-of-touch punditry, loony conspiracy theories. Much of this information is spread by people without skin in the game. (Or worse, people with incentives to lie.) And many people have lost faith in the experts and institutions, such as the government and the media, that once served as trusted referees.

Prediction markets, they believe, offer a better way to search for truth — rewarding those who are good at forecasting by allowing them to make money off those who are bad at it, while settling on the facts in an unbiased way.

(Continued in the article)
# The Handbook of Rationality (2021, MIT press) is now open access

Of particular interest here is section 4.1 on subjective accounts of probability and section 8.4 which explicitly addresses Knightian uncertainty and its challenges for Bayesian accounts.

https://direct.mit.edu/books/oa-edited-volume/5525/The-Handbook-of-Rationality
# Simple Terminal Colors

I work in the terminal a lot and I often want to write programs that color their output a bit. For example, perhaps I'm looking at a [sequencing read](https://www.jefftk.com/p/sequencing-intro) and I want to know what part of it was the reason it was flagged. I could use [colorama](https://pypi.org/project/colorama/), [termcolor](https://pypi.org/project/termcolor/), or another library, but [including a dependency](https://www.jefftk.com/p/designing-low-upkeep-software) for something this simple is not really worth it. Instead, printing colored text is as simple as:

COLOR\_RED\_BOLD = '\\x1b\[1;31m'
COLOR\_BLUE\_BOLD = '\\x1b\[1;34m'
COLOR_END = '\\x1b\[0m'
print(
  "One fish, two fish, " +
  COLOR\_RED\_BOLD + "red" +
  COLOR_END + " fish, " +
  COLOR\_BLUE\_BOLD + "blue" +
  COLOR_END + " fish.")

[![](https://www.jefftk.com/terminal-one-fish-two-fish-red-fish-blue-fish.png)](https://www.jefftk.com/terminal-one-fish-two-fish-red-fish-blue-fish-big.png)

I usually just want one or two highlight colors, typically in bold, and rarely find the six standard colors limiting. It's especially helpful if you want to draw your eye to something specific, while maintaining the context around it when you need to look further.

How compatible is this? Will someone with a non-ANSI terminal someday run my code and complain? One data point is that I wrote [icdiff](https://www.jefftk.com/icdiff) with this approach in [2009](https://www.jefftk.com/p/introducing-icdiff), and it's been reasonably popular. While I've gotten [hundreds](https://github.com/jeffkaufman/icdiff/issues?q=is%3Aissue) of bug reports this has never been one people have complained about.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0yNYkCYNaKotC3ZKBJZ2sCMiAogXFgBcjyfS9qW7JHduMXkUEBNxW5ywpcphoGJQel), [mastodon](https://mastodon.mit.edu/@jefftk/111207929256178296)*