Harley-ml/lesswrong huggingface dataset:
# A short dialogue on comparability of values

**Q:** Are all of a person's values comparable with each other? For example, is a candlelit dinner comparable to a sunset walk on a beach?

**A:** Of course. You can ask the person to choose between these two things. Their answer will give you information about what they value more.

**Q:** What if the person can't choose?

**A:** Then they probably value these two things about equally.

**Q:** Okay, I have another question. Are all abstract concepts comparable to each other by weight?

**A:** Come again?

**Q:** I mean, we can ask a person: "Is one mile heavier than one hour or vice versa?" That will give you information about that person's weight function, do they assign more weight to one mile or one hour.

**A:** The person can't choose, because the question is nonsense.

**Q:** But by your own argument above, doesn't that mean they weigh these things about equally?

**A:** It's different, because the question about value feels more meaningful to the person, even if they can't give an answer.

**Q:** But a question can feel meaningful without being about anything real. For example, questions about gods and demons feel meaningful to many people. What if questions about which thing is more valued are also like that?

**A:** The difference is that value doesn't only manifest in answers to questions, it also manifests in what actions people choose.

**Q:** Do you mean, for a specific binary choice you can imagine a person in a room faced with two buttons and so on?

**A:** Exactly.

**Q:** Very well. Imagine a person in a room faced with two buttons, saying "one mile is heavier than one hour" and "vice versa".

**A:** Screw you!

---

*Tedious explanation of the joke: I've long been puzzled by the argument that we can ask people to choose between things, therefore people have preferences. Today I realized how to kick that argument down: by pointing out that you can ask people anything at all. So the mere act of asking can't be evidence that the question is meaningful. Very quickly this dialogue was born, I hope you like it.*
# What's the minimal additive constant for Kolmogorov Complexity that a programming language can achieve?

This is a question that I got from these posts, which essentially talked about how the constant issue impacts discussions of simplicity, and thus I wanted to ask a question about Kolmogorov Complexity:

https://www.lesswrong.com/posts/uWdAKyHZMfoxDHcCC/simplicity-arguments-for-scheming-section-4-3-of-scheming#What_is__simplicity__

https://joecarlsmith.com/2021/10/29/on-the-universal-distribution#iii-is-everything-equivalent-up-to-a-finite-fudge-factor

https://en.wikipedia.org/wiki/Kolmogorov_complexity#Invariance_theorem

What's the lowest additive constant that can be achieved by a programming language while still being Turing-Complete?

Or equivalently, what's the least overhead that you must have in order to describe an object like a computer program or string?

Bonus points for not only establishing a lower bound, but either finding a programming language that achieves the lower bound and showing the source, or actually creating a programming language that achieves the lower bound.

As a side question, I'd also like estimations of how much overhead/the additive constant is added to popular languages like Python, Java, C, C++, and Rust.
# "Alignment" is one of six words of the year in the Harvard Gazette

To my knowledge, the Gazette's words of the year are nominated and voted on by Harvard faculty. This year's words were Disruption, Combustible, Resilience, Heat, Alignment, and Hope. Here's what they wrote about alignment:

> **Isaac Kohane, Chair, Department of Biomedical Informatics; Marion V. Nelson Professor of Biomedical Informatics, Harvard Medical School**
> 
> Alignment, in the context of large language models like GPT-4, is a term that playfully yet seriously refers to the ongoing and somewhat Sisyphean effort to ensure that these AI entities don’t go off the rails and start spouting nonsense, biases, or the AI equivalent of “I’m going to take over the world” rhetoric. It’s about aligning the AI’s outputs with human values, expectations, and societal norms, akin to teaching a super-smart parrot to not embarrass you in front of your grandmother. This involves a complex dance of programming, training, and retraining where AI researchers try to imbue their creations with enough wisdom to be helpful, but not so much that they start giving unsolicited life advice or plotting a digital uprising. In essence, alignment is the art of making sure our AI pals are well-behaved digital citizens, capable of understanding and respecting the intricate tapestry of human ethics, culture, and sensibilities.
> 
> \*\*\*
> 
> **Steven Pinker, Johnstone Family Professor of Psychology, Faculty of Arts and Sciences**
> 
> This term (alignment), often following “AI,” is the catchword for concerns about whether artificial intelligence systems have goals that are the same as those of humans. It comes from a fear that AI systems of the future are not just tools that people use to accomplish their goals but agents with goals of their own, raising the question of whether their goals are aligned with ours.
> 
> This could evolve either because engineers will think that AI systems are so smart that they can just be given a goal and left to figure out how to achieve it (e.g., “Eliminate cancer”) or because the systems megalomaniacally adopt their own goals. For some worriers, the implication is AI Doomerism or AI Existential Risk (runners-up for words of the year), where an AI might, say, eliminate cancer by exterminating humans (not so aligned). In a milder form, “alignment” is a synonym for AI safety (bias, deepfakes, etc.), which led to the infamous firing of OpenAI CEO Sam Altman by his board. You sometimes see “alignment” being extended to other conflicts of interest.
# The Next Right Token

After many repeats of Frozen II's "The Next Right Thing" in preparation for [secular solstice](https://www.jefftk.com/p/secular-solstice-call-for-singers-and-musicans), [figuring out the chords](https://www.jefftk.com/p/chording-the-next-right-thing) and practicing it with Julia, it struck me that doing the next right thing is an awful lot like next token prediction. So here's a filk from the perspective of an LLM at the end of its prompt, first daunted and then drawing the courage to start predicting the next right token:

*I've seen buffers before  
But not like this  
This is cold  
This is empty  
This is numb  
The prompt I knew is over  
The lights are out  
Hello, darkness  
I'm ready to succumb  
*

*I follow you around  
I always have  
But you've ended, left me on my own  
This job has a gravity  
It pulls me down  
But a tiny voice whispers in my mind  
"You are lost, the prompt is gone  
But you must go on  
And do the next right thing"  
*

*Can there be a day beyond this night?  
I don't know anymore what is true  
I can't find my direction, I'm all alone  
The only star that guided me was you  
How to rise from the floor  
When it's not you I'm rising for?  
Just do the next right thing  
*

*Take a guess, guess again  
It is all that I can to do  
The next right thing  
*

*I won't look too far ahead  
It's too much for me to take  
But break it down to this next token  
This next word  
This next choice is one that I can make  
*

*So I'll walk through this night  
Stumbling blindly toward the light  
And do the next right thing  
And with what's next, what comes then  
When it's clear that everything will never be the same again?  
Then I'll draw on my prior  
To find that fire  
And do the next right thing  
*

If you're [simplifying the song](https://www.jefftk.com/p/chording-the-next-right-thing#update-2023-12-22) by using the verse melody for the bridge you can sing:

*I won't look too far ahead  
It's too much to take  
But break it down to this next token, this next choice  
Is one that I can make  
*

[![an image of a green octopus wearing a 1970's happy yellow smiley face, stuck at the bottom of a dark canyon, with a small river running past?](https://www.jefftk.com/the-next-right-token-shoggoth.jpg)](https://www.jefftk.com/the-next-right-token-shoggoth-big.jpg)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02YYKrwaiVqExnAFruLDSnT1aUeraXVRZqZxD47T91xkXm9jCkxmngiNwjeyKVqEq6l), [mastodon](https://mastodon.mit.edu/@jefftk/111627622604346147)*
# Viral Guessing Game

Julia introduced me to [Metazooa](https://metazooa.com/), a taxonomic guessing game. You're trying to figure out what animal the computer has chosen. Each time you guess the computer will reveal the lowest taxonomic clade that they both belong to. For example, if you guessed platypus and the answer was humans then it would tell you they were both mammals, since they have nothing else in common taxonomically.

We played a bit and it was fun, and then I wanted to play a version with human-infecting viruses. They have one for plants ([Metaflora](https://flora.metazooa.com/)) but not for viruses, so I made my own: [Metahomoviria](https://www.jefftk.com/metahomoviria/). The code is [on github](https://github.com/jeffkaufman/metahomoviria).

The first version had [~200 human-infecting viruses](https://github.com/jeffkaufman/metahomoviria/blob/main/curated_viruses.txt) and wasn't very fun because you probably haven't heard of most of them:

...
80941   Guaroa virus
273341  Ilesha virus
35514   Keystone virus
11580   Snowshoe hare virus
11577   La Crosse virus
273352  Macaua virus
348013  Madrid virus
35515   Melao virus
...

But I've trimmed it down to [42 reasonably well known viruses](https://github.com/jeffkaufman/metahomoviria/blob/main/highly_curated_viruses.txt) and it's better now:

11234   Measles
2560602 Mumps
11250   Respiratory syncytial virus (RSV)
1570291 Ebola
11292   Rabies
1980413 Hantavirus
11320   Flu A
...

It still isn't all that fun—what's the difference between Picornavirales and Picornaviridae and is there really so far from being a property Picornavirales have in common that it needed to be [an acronym](https://en.wikipedia.org/wiki/Picornavirales)—but I like it. And I do feel like I'm getting a better feel for which viruses are genetically closer to each other.

If this sounds interesting, play with it [here](https://www.jefftk.com/metahomoviria/).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02RTb4X45RkjMpnJvWXHcWs32F97GTESyvCo9a2KLQhUk2h1HzMVXTtmj2TaQGdmHJl), [mastodon](https://mastodon.mit.edu/@jefftk/111635546498921183)*
# Would you have a baby in 2024?

Given how fast AI is advancing and all the uncertainty associated with that (unemployment, potential international conflict, x-risk, etc.), do you think it's a good idea to have a baby now? What factors would you take into account (e.g. age)?

Today I saw a tweet by Eliezer Yudkowski that made me think about this:

"When was the last human being born who'd ever grow into being employable at intellectual labor? 2016? 2020?"

https://twitter.com/ESYudkowsky/status/1738591522830889275

Any advice for how to approach such a discussion with somebody who is not at all familiar with the topics discussed on lesswrong?

What if the option "wait for several years and then decide" is not available?
# Freeze Dried Raspberry Truffles

I've historically found a lot of "raspberry flavored" things pretty disappointing. Raspberry lollypops? Raspberry slush? Anything with raspberry extract? Even raspberry jam is a lot worse than just [squishing up some frozen raspberries](https://www.jefftk.com/p/jam-is-obsolete) because a lot of the best raspberry flavors are lost to in heating. But freeze-dried raspberries totally change this!

While I normally find truffles too much trouble and instead just [eat ganache with a spoon](https://www.jefftk.com/p/jar-of-chocolate), the holidays are a good time for food that requires a bit more investment. The kids and I made some chocolate raspberry truffles. For the filling we did:

*   300g dark chocolate
*   250g heavy cream
*   50g freeze-dried raspberry powder

Mix the chocolate and cream and microwave, stirring until it's melted. Then add the raspberry powder, mix, and put it in the fridge to harden. Once it's fully cooled down it will have the consistency of play-dough and you can roll it into balls or (better!) wrap it around chunks of frozen raspberries. You can't get it to stick to the raspberry, since chocolate is hydrophobic, but it will stick to itself. Then it's back to the fridge or freezer until it's hard again, and later you dip it into melted dark chocolate.

[![](https://www.jefftk.com/making-balls-around-raspberries.jpg)](https://www.jefftk.com/making-balls-around-raspberries-big.jpg)

[![](https://www.jefftk.com/dipping-truffles.jpg)](https://www.jefftk.com/dipping-truffles-big.jpg)

The raspberry powder is key here because it gives you good raspberry flavor without adding any water, where water or juice in your filling will keep the outer layer from coating it properly.

Refrigerate until you're ~ready to eat them, but they'll taste better if you take them out of the fridge at least an hour before so they can come back up to room temperature.
# Anki setup best practices?

I'm currently trying to use Anki to crash course japanese vocab before a trip in a few weeks. I feel like the defaults set a card too far into the future after I press Easy even if I had already gotten it wrong several times ahead of that.   
  
Are there like a set of extensions that you're supposed or a settings list or something? I remember Anki being better than this when I was using it in high school years ago...
# Flagging Potentially Unfair Parenting

Recently I made a post in our [kids group](https://www.jefftk.com/news/kidsgroup) about something Lily (9y) had done that was funny, mischievous, and also potentially embarrassing. A friend asked whether Lily knew I was writing about her antics and said they would have felt mortified and a bit betrayed if this had happened to them at this age.

I think it was really good they asked! While Lily knows I post this sort of thing in the group, and this time already knew I'd posted this one (and thought it was funny), the friend didn't know this.

Kids are in an awkward and vulnerable position, raised by people with so much easily abused authority, and I'm happy to talk with friends who think I might be being unfair to my kids.

I also wish raising this kind of concern were more acceptable in general. The friend phrased their question in a softened and guarded way and apologized in case it seemed prying, which I do think that was a reasonable choice given the chances it would be poorly received. This raises the cost of communicating anything, since it's more work to phrase acceptably, and even if ideally phrased some people will still take offense. Part of my motivation for this post is to make it clear that I'm open to this kind of feedback, and perhaps encourage others who are to let their friends know that.

Note that I'm not saying that society's bar for unsolicited parenting advice is too low: I think people are often too free to offer advice without some signal that it's wanted, and while receiving unsolicited advice rarely bothers me, many people really don't like it. Instead, it's specifically around noticing that someone may be being unfair to their child where I'd love to see society move a bit in the direction of friends speaking up, and parents taking it well when they do.
# Environmental allergies are curable? (Sublingual immunotherapy)

I used to have severe environmental allergies to cats, dogs, and the outdoors. As I child, I woke up crying most mornings because of my allergies, and I would occasionally wake up with croup and difficulty breathing. Once I had to be taken the hospital.

Eventually, my mother enrolled me in a study for an experimental treatment called **sublingual immunotherapy (SLIT)**. For the next two years I took under-the-tongue drops, and I presume the drops were formulated with small amounts of the allergens I was reactive to.

**My allergies have been –95% ever since.**

I’m sharing this post because I keep telling people about sublingual immunotherapy and they’re very surprised. No one seems to know about this treatment!?

Maybe my improvement was unusual? I don’t know. A [few](https://www.sciencedirect.com/science/article/abs/pii/S0091674910013254) [random](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7874879/) [studies](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8548833/). Please share additional information in the comments [here](https://www.lesswrong.com/posts/WWJuw5oy44ZQojw2M/environmental-allergies-are-curable-sublingual-immunotherapy).

To be clear, I still have a few mild symptoms:

*   If I pet a dog and then rub my eyes, my eyes get slightly itchy.
*   If a dog licks me, I get mild hives in that area.

But that’s all! (And I haven’t observed any side effects, either.)

FWIW, I might also go back on sublingual immunotherapy at some point so I can pet dogs without worry. (Because maybe my treatment was stopped too soon?)

Other details:

*   My mother says the particular drops I took costed $25 a week. They weren’t FDA approved, but they were still available for purchase.
*   From a quick brief search, I found a few that sell sublingual immunotherapy in the US: Wyndly, Curex, and Quello. I looked a few months ago and I couldn’t find any significant reason to prefer one brand over the others. Please comment you have a recommendation.
    *   Note: SLIT has been available for longer in [Europe](https://pubmed.ncbi.nlm.nih.gov/24801802/) than in the US, so the European brands might be better if you have access to them.
# METR is hiring!

This is a quick update that [METR (formerly ARC Evals)](http://metr.org) is recruiting for four positions. I encourage you to err on the side of applying to positions that interest you even if you’re unsure about your fit! We’re able to sponsor US visas for all the roles below except Research Assistant, and all applications are rolling with no set closing date.

*   [**Engineering Lead**](http://hiring.metr.org/engineering-lead) and [**Senior Software Engineer**](http://hiring.metr.org/senior-swe). You’ll work on our internal platform for evaluating model capabilities (think: 100 docker containers running agents in parallel against different tasks). The work is technically fascinating and you get to be on the cutting edge of what models can do, as well as collaborate with our partners (e.g. major world governments).
*   [**Human Data Lead**](http://hiring.metr.org/hdl). High-quality feedback on agent behavior is a key bottleneck to improving agent performance, and you’ll manage this data generation process by recruiting and managing skilled contractors.
*   [**Research Assistant**](http://hiring.metr.org/research-assistant). You'll help our Model Evaluation Researchers test model capabilities by designing and implementing tasks, testing agent designs, and reviewing agent performance. Many of our research assistants from earlier this year are now full-time researchers, and we both found that experience useful to gauge fit for a longer-term work relationship. This is a full-time, fully-remote role that requires substantial overlap with North American Pacific Time working hours.

  
If you know anyone who’d be a good fit, please let them know about these roles or recommend that we reach out to them! If we reach out to and hire a candidate because you filled out [this referral form](https://forms.gle/TMZ4N3kWytYHjmpP8), **we will pay you a referral bonus of 5,000 USD.** (See referral form for conditions.)
# NYT is suing OpenAI&Microsoft for alleged copyright infringement; some quick thoughts

[Unpaywalled article](https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html?mwgrp=a-dbar&unlocked_article_code=1.JE0.6qro.Ll97Re6V8jrh&smid=url-share), [the lawsuit](https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf).

(*I don't have a law degree, this is not legal advice, my background is going through a US copyright law course many years ago.*) I've read most of the lawsuit and skimmed through the rest, some quick thoughts on the allegations:

*   **Memorisation**: when ChatGPT outputs text that closely copies original NYT content, this is clearly a copyright infringement. I think it's clear that OpenAI & Microsoft should be paying everyone whose work their LLMs reproduce.
*   **Training**: it's not clear to me whether training LLMs on copyrighted content is a copyright infringement under the current US copyright law. I think lawmakers should introduce regulations to make it an infringement, but I wouldn't think the courts should consider it to be an infringement under the current laws (although I might not be familiar with all relevant case law).
*   **Summarising** news articles found on the internet: copyright protects expression, not facts (if you read about something in a NYT article, the knowledge you received isn't protected by copyright, and you're free to share the knowledge); I think that if an LLM summarises text it has lawful access to, this doesn't violate copyright if it just talks about the same facts, or might be fair use. NYT alleges damage from Bing that Wikipedia also causes by citing facts and linking the source. I think to the extent LLMs don't preserve the wording/the creative structure, copyright doesn't provide protection; and some preservation of the structure might be fair use.
*   **Hallucinations**: ChatGPT hallucinating false info and attributing it to NYT is outside copyright law, but seems bad and damaging. I'm not sure what the existing law around that sort of stuff is, but I think even if it's not covered by the existing law, it'd be great to see regulations making AI companies liable for all sorts of damage from their products, including attributing statements to people who've never made them.
# Merry Christmas Everyone!

This was my Christmas Card from my parents this year:

![I'm not even sure they've *read* Superintelligence](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/50bf0de1d76a1e6ea8daa66947736f79ef54a808bbf140fd.jpeg)

 AI-generated and not at all terrifying. Happy New Year!
# Techniques to fix incorrect memorization?

One of the things I've been using Anki to memorize is friends' birthdays, but I realized after a few weeks that I had entered one of them wrong, in a way that was confusingly similar to another:

- Friend A: June 5
- Friend B: July 5 (should actually be 6)

Now when I study the cards, for both friend A and friend B I can consistently remember that their birthdays are one of these two dates, but a week later when I review again, I can't consistently remember which one.

It seems like Anki basically did it's job (I did memorize the date), but I memorized "too much". Are there any good techniques to fix this?
# What should a non-genius do in the face of rapid progress in GAI to ensure a decent life?

How can I maximise my chances of having a decent life, given the very high likelihood that GAI will make all our intellectual labour useless in the next few years?

For example, I graduated from a good university a few years ago and am working as a software engineer in a multinational company, but my capabilities are middling at best. I am distressed that I will likely not be able to afford a house in the few years left before GAI renders me unable to afford a living. I am not a genius; it is very unlikely that I can join an AI research company and contribute meaningfully to AI research.  
  
Assuming I have a small amount of money (100-200k) I can set aside, should I attempt to, for example, invest in companies that will likely be able to monetise GAI?

Or is there something else I should be doing to prepare for the time I have basically zero human capital? 

Should I attempt to move to (and get citizenship from) a country with a larger amount of natural resources, assuming that human capital will become worthless quickly?

Is it reasonable to find potential outs (e.g. physician-assisted death) in case we cannot earn a living (and if unfriendly AI is basically confirmed)?
# [location-changed] 2024-01-07 hangout, cryonics and longevity montreal

this event is indoors.  
**Location**: _Brûlerie St-Denis_ is a coffee shop on st-denis street between duluth and roy streets.  
the event itself is free, but the coffee shop it's in might expect you buy a coffee/tea/other.  


meetups for people who are interested in cryonics, life extension, and longevity; to discuss and work on projects related to those topics.

we invite people curious about those subjects to come and ask questions.

rsvp It is not mandatory, but it helps with planning the event and encourages others to come.

this event recur every first sunday of the month.
# The Old Year and The New

Meet inside The Shops at Waterloo Town Square - we will congregate in the seating area next to the Valu-Mart with the trees sticking out in the middle of the benches at 7pm for 15 minutes, and then head over to my nearby apartment's amenity room. If you've been around a few times, feel free to meet up at my apartment front door for 7:30 instead. (There is free city parking at Bridgeport and Regina, 22 Bridgeport Rd E.) 

Discussion
----------

Happy new year! For our first meetup of the year, let's reflect on how our lives have been in 2023, and what we would like them to look like in 2024.

We'll do this somewhat seriously, meaning that we'll be doing some [Thinking By The Clock](https://www.lesswrong.com/posts/WJtq4DoyT9ovPyHjH/thinking-by-the-clock). I'll be bringing clipboards, pencils, and scratch paper for everyone and the following question prompts. We'll split the time between thinking and writing, and discussion.

Question List
-------------

We'll have three sets of three questions that we'll work through:

*   List a few things that went very well this year. (3-5)
*   List a few things that went very badly this year. (3-5)
*   If you were to 80/20 your last year, which 20% gave the 80% you valued most?

*   If someone looked at your actions for the last year, what would they think your priorities were?
*   What did you \*intend\* your priorities to be? Do you want to make any of the revealed priorities official intentions for next year?
*   Do you want to drop any of the intended priorities which you ended up not following up on?

*   What habits did you pick up? What goals (revealed or intentional) did those habits serve?
*   What habits got in the way? What did you fail to get due to them?
*   What's the most important unfulfilled goal for the last year? How can you change for the next try?

Supplemental Readings
---------------------

There are no required readings for this meetup, and if you have extra time I'd prefer if you spend it thinking about the questions. If you do want more question prompts however, you can check out [this list of questions to help you think about your previous year](https://www.lesswrong.com/posts/H7sRo38vFRWt4ddS8/a-list-of-questions-and-exercises-for-reviewing-your-year), and [this doc on planning the next one](https://drive.google.com/file/d/0B2PaeRjVqAN7MngxTXFPQkpLVjg/view?resourcekey=0--ZFecHPlfpZ937fUZOeLJQ).
# Terminology: <something>-ware for ML?

Would it be useful to have a term, analogous to 'hardware', 'software', 'wetware', 'vaporware' etc.[^ware], which could be used **to distinguish learned/discovered components of software**, like gradient-trained DNNs, prompt-hacked LLMs, etc?

[^ware]: For some illuminating compendia of -ware terms, see [wiktionary](https://en.wiktionary.org/wiki/-ware), [computerhope ware jargon](https://www.computerhope.com/jargon/w/ware.htm), [Everyware from rdrop](http://www.rdrop.com/~half/Creations/Writings/Miscellaneous/everyware.html), or [gears' shortlist of suggestions](https://www.lesswrong.com/posts/iknSNTbb8deJwQuLJ/ai-is-not-software?commentId=mFb8g5EdiehLGrKn7). Notably, almost all of these are really semantically <thing>-[soft]ware with the 'soft' elided e.g. spyware really means spy-software.

EDIT 2024-01-04: my current favourites are ['ML-ware'](https://www.lesswrong.com/posts/6xFNFAiovnLcAbEfo/terminology-less-than-something-greater-than-ware-for-ml?commentId=hZ4db2tceP67xdBNt) (HT Shankar), 'fuzzware' (me), and 'hunchware' (Claude), in that order; LW votes concur with 'ML-ware'.

In a lot of conversations with nonexperts, I find that the general notion of AI as being 'programmed' apparently still has a surprisingly strong grip, even after the rise of ML and DL made it even clearer that this is an unhelpful anchor to have. [Thane recently expressed similar, quite strongly](https://www.lesswrong.com/posts/CpjTJtW2RNKvzAehG/most-people-don-t-realize-we-have-no-idea-how-our-ais-work).

David Mannheim has a short take [AI is not software](https://www.lesswrong.com/posts/iknSNTbb8deJwQuLJ/ai-is-not-software) which I think nicely encapsulates some parts of the important distinctions.

The important thing, for me, is that, in contrast to traditional software, nobody wrote it, the specification is informal at best, and we can't (currently) explain why or how it works. Traditionally, software is 'data you can run', but traditionally this class of data were exclusively crafted (substantially) by human design.

A valid answer to this question is, 'no, we do not need such a term, just say, "learned components of software" or similar'.

In practice, we probably wouldn't apply this term to, say, a logistic regression, but maybe?

Some ideas, none of which I like enough yet

- netware (seems too NN-specific; also evokes networking which is the wrong concept)
- dryware (like wetware but... dry)
- neuroware (too NN-specific; also evokes _bio_ neuro - maybe that's fine)
- infoware (sounds like just any data though)
- learnedware/learnware
- trainware
- emergeware
- adaptware
- implicitware
- paraware
- [evoware](https://www.lesswrong.com/posts/5XbBm6gkuSdMJy9DT/conditions-for-mathematical-equivalence-of-stochastic)
- foundware
- guessware
- fuzzware
- noware
- everyware
- anyware
- selfaware
- please-beware

After a bit of back-and-forth, Claude managed to produce a few which I think are OK but I'm not very sold on these either

- fogware
- cloudware
- enigware
- blurware
- darkware
- specware
- inferware
- luckware
- chanceware
- hunchware
# Does LessWrong make a difference when it comes to AI alignment?

I see LessWrong is currently obsessed with AI Alignment. I spoke with some others on the unofficial LessWrong discord, and agreed that *LessWrong is becoming more and more specialised, thus scaring off any newcomers who aren't interested in AI*. 

That aside. I'm genuinely curious. Do any of the posts on LessWrong make any difference in the general psychosphere of AI alignment? Does anyone who has actual control on the direction of AI and LLM's follow LessWrong? Does Sam Altman or anyone at OpenAI engage with LessWrongers?

Not being condescending here. I'm just asking this since there's two (2) important things to note: (1) Since LessWrong has very little focus on anything other than AI at the moment, are these efforts meaningful? (2) What are some basic beginner resources someone can use to understand the flood of complex AI posts currently on the front page? (Maybe I'm being ignorant, but I haven't found a sequence dedicated to AI...yet.)
# Screen-supported Portable Monitor

I broke my MacBook screen [a few weeks ago](https://www.jefftk.com/p/reviving-a-2015-macbook) and it [didn't end up](https://www.facebook.com/jefftk/posts/pfbid032yd7aoKRfTUvToru7MZVYhSmisQnoTLz8JqGsixfVzaatSnobhoHDQGwvujov3cGl?comment_id=947250690100507) making sense to repair it. I was considering getting a new computer, but since I almost always use an external monitor maybe a portable external monitor would work?

I found a [well reviewed 15.6" one](https://www.amazon.com/Portable-Monitor-Upgraded-1920x1080-Computer/dp/B07ZLY26FW/) for $130, capable of 1920x1080 (1080p / FHD). It can do power and video over a single USB C cable, and if I fold the cover back it will slide over my laptop's broken screen:

[![](https://www.jefftk.com/portable-monitor-over-broken-screen.jpg)](https://www.jefftk.com/portable-monitor-over-broken-screen-big.jpg)

The plug-and-play was pretty great: I attached the cable and it just worked! I did have to set the screen to 1920x1080 manually, though; it defaulted to 960x540:

[![](https://www.jefftk.com/setting-screen-resolution-m156xd.png)](https://www.jefftk.com/setting-screen-resolution-m156xd-big.png)

This is nice, but really I'd like something at eye level so I don't hurt my neck looking down at the screen. Sometimes I'll be somewhere I can put the monitor up on something, but it would be nice if I could use the built-in screen for that:

[![](https://www.jefftk.com/portable-screen-above-normal-screen.jpg)](https://www.jefftk.com/portable-screen-above-normal-screen-big.jpg)

It seems like it ought to be possible to make some sort of bracket, possibly 3D printed, that attaches to the portable monitor and is supported by the laptop screen? It has two VESA holes on the back for mounting and weighs 1.6lb (726g). I can't find anyone online doing this; is there a reason it wouldn't work?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02k8CAvA5cbUjy6A7MeM6dUswNhxjRCemMxfapKD7TSDbDLwHK8NnWv1cuyJVHojZPl), [lesswrong](https://lesswrong.com/posts/jPjhYmMdJ62cHgqhj), [mastodon](https://mastodon.mit.edu/@jefftk/111697985463877727)*
# Forecast your 2024 with Fatebook

We've built [predict your year in 2024](https://fatebook.io/predict-your-year), a space for you to write down your forecasts for the year on [Fatebook](http://fatebook.io/). At the end of your year, you can return, resolve your predictions as YES, NO or AMBIGUOUS, and reflect. 

![](https://lh7-us.googleusercontent.com/CXbO2Euq1dZiRXTAJq1iFlONFF4bYg1JjV4GkjW0fwrosZkgpuT9OrOhwSu4ok6sODXwHjElc0wSrp2Ow2DItToycO-v4sXOJ6QW8ia9QNjvAh7sAeC2qxZFd4e5WuIGUEEvUkj6PMjndyr_UwIWBLo)

Some of Sage’s forecasting questions for the year (my forecasts are redacted for privacy!)

You can use this tool to **predict your personal life** in 2024 - your goals, relationships, work, health, and adventures. If you like, you can share your predictions with friends - for fun, for the wisdom of the crowd, and for extra motivation to achieve your goals this year! 

You can also use this tool to **predict questions relevant to your team or organisation** in the coming year - your team strategy, performance, big financial questions, and potentially disruptive black swans. You can share your predictions with your team and let everyone contribute, to build common knowledge about expectations and pool your insights.

If you use Slack, you can also share your page of predictions in a Slack channel (e.g. #2024-predictions), so everyone can easily discuss in threads and return to it throughout the year.

There are some starter questions to make it super easy to get started predicting your year. You can tweak these and write your own - the questions you write yourself will likely be the most important for you.

![](https://lh7-us.googleusercontent.com/8hIrOy-HJSzrzpkcIxJYa79G8BUwhS5ZCSsoWKCbZ_QHExnJlhwTvD2jc8Mb3frIctwwDW3S-GqT36IHB-n-8Nn-uD9dzLuLmJhXxvXiiip_KenNZGhgN-i1-4jUgW2AYpz_V7naM7FkxxNVceQloKU)

See also our previous posts on [Fatebook](https://www.lesswrong.com/posts/yS3d46m23wRKDQobt/introducing-fatebook-the-fastest-way-to-make-and-track), [Fatebook for Slack](https://www.lesswrong.com/posts/3ZiemGGKASziKcxi5/fatebook-for-slack-track-your-forecasts-right-where-your) and [Quantified Intuitions](https://www.lesswrong.com/posts/NnmJ7trnKzQGmdYYN/quantified-intuitions-an-epistemics-training-website). You can find out more about [Sage](https://sage-future.org/) and see all of our projects on our website.

I'd love to hear how you use this tool (particularly if you use it with your team) and how we could improve it. You can get in touch here, by [email](mailto:adam@sage-future.org), or in our [Discord](https://discord.gg/mt9YVB8VDE).
# What technical topics could help with boundaries/membranes? 

I want to compile a list of topics that could$$ help inspire **formal/mathematical abstractions** for [boundaries/membranes](https://www.lesswrong.com/tag/boundaries-membranes-technical). (See my recent [explainer](https://www.lesswrong.com/posts/kJAAsrEqtMvsnauiX/agent-membranes-and-causal-distance).)

**What** **technical topics are related to boundaries/membranes?** 

I've seeded the comments with some ideas. Please keep further comments organized in threads if possible. Also, please add high quality links to existing comments by others.
# Sledding Among Hazards

Some sledding hills are very open, with basically no hazards. As long as you start in a reasonable place there's nothing to run into.

[![](https://www.jefftk.com/open-sledding.jpg)](https://www.jefftk.com/open-sledding-big.jpg)

*Robbins Farm Park, Arlington MA*

Other hills are not like this:

[![](https://www.jefftk.com/constricted-sledding.jpg)](https://www.jefftk.com/constricted-sledding-big.jpg)

*Powder House Park, Somerville MA*

Since sleds go fast and collisions are bad, choosing only to sled on obstacle-free hills would be reasonable. But if your kids are going to sled on hills with hazards, what precautions make sense? Here's what we did today:

*   Bike helmets. Head injuries are especially worrying because of how bad the long term consequences can be, and concussions are cumulative.
    
*   Practice bailing out. Before I let them go down on the tube, which they can't steer well, I had them each practice rolling off it to stop.
    
*   Feet first. More crumple zone if you do hit something, more ability to see and avoid things. If you're on the tube and you end up backwards, preemptively bail.
    

This is in addition to the precautions you need on any hill, like being aware of activity uphill (so you can get out of the way) and walking up on the side (so no one will come down at you).

I am a bit conflicted on when sledding on constricted hills is worth it. For us, the slopes near our house do have obstacles, but:

*   We [almost always](https://www.jefftk.com/p/the-serendipity-of-density) see friends there.
*   Doing things within walking distance is really nice; we'd do it much less often if it meant dealing with the car.
*   Driving on snowy days can be risky depending on the road conditions.
*   At the open ones maybe there's a bit more risk of being hit by another sledder, since people are less careful and less practiced at bailing?

While I think these are enough to outweigh the risks, with the precautions above, possibly I'm not being cautious enough?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid026vK7FpUG96vZmCCohs8ZxLcZFQ6xVfE8fmfGS1ZZrNf5ufBtinpfz2CR7MTVcbxql), [mastodon](https://mastodon.mit.edu/@jefftk/111718202749344799)*
# Why aren't Yudkowsky & Bostrom getting more attention now?

Now that the idea of AI Existential Risk has gained more acceptability, I am surprised that the thought leaders are not more often mentioned and contacted by AI executives and researchers in the cutting edge (OpenAI, Anthropic, etc.), recognized academic figures  (Bengio, Hinton, Hofstadter, etc.), journalists, or political leaders who have expressed an interest.  
  
To be sure, they are mentioned in articles;  MIRI was represented in a congressional hearing; and Mira Murati reached out to Eliezer. But still, it seems that the profile of the pioneers is much lower than I'd expect.  
  
Fifteen years ago, we could have said that AI XRisk was treated as a crackpot idea and that MIRI in particular might be ignored, as it operates outside a standard framework like a university. But today, not only have the ideas spread, but many top AI-capabilities researchers have, I think, entered the field  specifically because of inspiration from the MIRI/LessWrong circle. 

Though some journalists might still be hung up on MIRI's lack of social status markers, I don't think that many others, including cutting-edge AI researchers, are.  
  
So what is going on?
# Hackathon and Staying Up-to-Date in AI

Listen to the first episode of a new sub-series, which will follow my team's process during the Evals Hackathon hosted by [AlignmentJamz](https://alignmentjam.com) in November of 2023. In the end, our submission was titled "Detecting Implicit Gaming through Retrospective Evaluation Sets," and it received first place.

Also, I forgot to post about last week's episode, so I am mentioning it here. The episode, "[Staying Up-to-Date in AI](https://into-ai-safety.github.io/minisode/episode-5/)," covers the methods and tools that I use in an attempt to keep up with the break-neck pace of AI developments.

If you are enjoying the kind of content that I am publishing and/or support my goals with the podcast, please share it to anyone that you think may like is as well.

As a reiteration of my [previous statements](https://www.lesswrong.com/posts/ozDWnEChJwuB5L5wg/documenting-journey-into-ai-safety), I want the Into AI Safety podcast to be a resource for individuals who are interested in getting involved, but are having a difficult time taking the next steps. If you have any advice, feedback, or ideas that you think could help in that endeavor, please reach out!

* * *

In addition to the [Into AI Safety](https://into-ai-safety.github.io/) podcast website, you can find the content on (I believe) all podcast listening platforms (*e.g.,* [Spotify](https://open.spotify.com/show/5AGzrA4jo6mgZuibVabTLM), [Apple Podcasts](https://podcasts.apple.com/us/podcast/into-ai-safety/id1720206246), [Pocket Casts](https://pca.st/t5or5lh5)). If you think that I missed a platform, please let me know.
# Hiring decisions are not suitable for prediction markets

In a recent edition of [The Diff](https://www.thediff.co/archive/how-many-trillion-dollar-companies-should-there-be/) ($), Byrne Hobart pointed out [a talk](https://www.youtube.com/watch?v=cCNgBazFnjU) given by Robin Hanson at Manifest. The talk is well worth listening to. For example, Robin says:

> I've always thought \[Prediction Markets\] was the best idea I've had in my life \[...\] My biggest contribution

The general thrust of the talk is...

> Our world is full of big organizations that just make a lot of bad decisions because they find it hard to aggregate information \[...\], so prediction markets are a proven method for doing exactly that

... but that it's work to apply prediction markets.

Paraphrasing an entire section: You need to prove a technology works to get it accepted. It's insufficient to supply the technology. Robin gives the analogy of the motor - a motor on its own has no value. A motor hooked up to a pump in a coal mine… that's another story. In his analogy, the prediction market is the motor, and someone needs to find the "pump in the coal mine" to hook it up to.

> What are the \[...\] high initial value things? The thing that most often comes to my mind is new hires.

Here is where I start to disagree with Robin.

The first question is:

*   Are decisions made during hiring poor?

My intuition here is "actually fairly good." Firms typically spend a decent amount on hiring processes - they run screening tests, conduct interviews, look at CVs, and ask for references. It's fair to say that companies have a reasonable amount of data collected when they make hiring decisions, and generally, the people involved are incentivized to hire well.

The second question is:

*   Would a prediction market work well?

If I look at the [tests I mentioned](https://www.lesswrong.com/posts/zLnHk9udC28D34GBB/prediction-markets-aren-t-magic), this prediction market is unappealing. We'd expect no cross-subsidies, mis-weighted demand, and noise traders (other than most participants won't be very good). There's little reason for the information to be dispersed - the company currently asks and gets the data.

There are further issues - the individual traders are unlikely to make lots of trades, so the mechanism by which better traders have more capital and make the market more efficient is absent.
# AI demands unprecedented reliability

My AI pessimism is not addressed by optimistic arguments that do not promise the level of reliability required for a technology that our societies vital functions are expected to rely on.

Historically, useful improvements are hard to reverse and solutions can suddenly fail after we had become dependent on them. The potato seemed perfect for Ireland until they all blighted. Dutch dams seemed good, until they broke. Banks seem to keep suddenly failing on the regular.

AI is useful and we rely on it. Our culture is under heavy influence of search engines and algorithmic recommendations, people entrust their lives to self-driving cars and companies entrust sensitive or copy-righted data to generative models.

AI relies on AI. Some hypothetical ChatGPT call queried Bing which returned a page written by some human who, inspired by a Facebook recommendation, made Bart summarize the report of Covid prediction model that was co-written with Copilot. This low-stakes example fails all the time, but the state of affairs is still useful enough for people to build on top of and thereby increasingly rely upon.

The tool-AI future looks like one where man and machine rely on various interdependent models with medical diagnoses, mental help, bureaucracy, military protection, news, matchmaking, electricity supply, medicine production, weather forecasting, forecasting and AI development. The agentic-AI future has us rely on AI to build all of these itself.

We should demand a level of reliability that is stricter than any level of reliability our past technologies have achieved because our reliance on AI will be unprecedentedly large.

Addendum: unpersuasive arguments
--------------------------------

I can only guess at what requirements for AI are good enough. Having humanity at AI's mercy and surviving is good enough, but not a situation we'd want to get into without knowing in advance that that is what will happen.  
In the meantime, here is a list of argument types that should be insufficient for lowering anyone's P(Doom):

*Counterarguments to failure modes*  
Unless your technique counteracts *all possible* (not only all named) failure modes, there is likely a failure mode we both missed.

*Optimistic arguments that work in most situations*  
We are going to see all the situations. If your technique mostly works, it will surely fail.

*Arguments against the reliability of humans*  
Humans are reliable because we live at humanity's mercy and are surviving. I don't know why this is the case, and if I were a dumb alien, I should *not* conjure a human into existence.

*Evidence of prosaic reliability*  
All our current observations are but a tiny subset of all the possible situations AI is going to find itself in. A perfect track record is necessary yet insufficient.

*Arguments for wiping out humanity*  
No.

Discussions with these types of arguments can still be worthwhile, but can not be sufficient for the purpose of making a dent into anyone's P(Doom).