Harley-ml/lesswrong huggingface dataset:
# How can we secure more research positions at our universities for x-risk researchers?

Has anyone had experience with this? Is anyone working on this?  
  
Reading [this](https://www.alignmentforum.org/posts/4jFnquoHuoaTqdphu/ai-x-risk-reduction-why-i-chose-academia-over-industry) and [this](https://www.alignmentforum.org/posts/HXxHcRCxR4oHrAsEr/an-update-on-academia-vs-industry-one-year-into-my-faculty) give me reason to think that it could be more impactful for some x-risk researchers to have research positions at universities.

However:

*   Universities are financially constrained with regard to the number of researchers they can hire
*   When they *can* hire, they rarely select candidates based on their expected contribution to safeguarding future lives
*   Successful candidates are immediately tasked with non-research obligations which take up a substantial proportion of their time

Can we remove these hurdles? Here is my proposal. **A new or existing EA org establishes agreements with university departments/schools whereby:**

1.  The org decides on which researcher to hire (with the department having the ability to veto)
2.  The org funds the researcher
3.  The department treats the researcher as one of its own, except without the non-research obligations

I do recognise that currently, university researchers can and do get EA funding which enables them to focus on their high-EV research. An example that comes to mind is Arif Ahmed at the University of Cambridge who, in 2019, received funding from CLR. Additionally, there are research institutes which are well-integrated at universities (e.g. GPI at the University of Oxford). Are these models better? Could my proposal still be impactful and worth pursuing?

If anyone has any experience with or thoughts on this, please comment below.
# Is training data going to be diluted by AI-generated content?

This is my first post on here so please be lenient if I fail to follow any norms.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/baaa02936ba2e09ee2022896faae53dda7eab2d50c9ccaa0.jpeg)

An image made by Midjourney AI. The prompt was "the physical manifestation of logic"

  
With the explosion of AI-generated images and text from Dall-E 2, Midjourney AI and GPT3, it does not seem unreasonable to assume that a non-negligible part of the content of the internet might become AI-generated. This in itself is not problematic. I actually look forward to the internet where everyone has access to the means to create awesome media for almost nothing.   
But, since most large language models and multimodal AIs are trained on a dataset that basically consists of "all of the internet", we could end up with a feedback loop. If a lot of the old model's output is used as part of the training data for a new model, the development will slow down and the new models will essentially emulate the old ones. Maybe I'm underestimating the amount of content on the internet or overestimating the use of AI. This would of course also only occur if the development of AI is slow enough for the Internet to "fill up" with AI-generated content.  
And sure we could say that we only use human-generated content for the training data, but since we're literally trying to make AI-generated content indistinguishable from human-generated content, we are actively working against ourselves and if we have any success we won't have success. 

  
Essentially my question is "is training data dilution going to be a thing?"
# Galaxy Trucker Needs a New Second Half

I like [Galaxy Trucker](https://boardgamegeek.com/boardgame/31481/galaxy-trucker) quite a bit. It has two very different halves:

1.  Everyone grabs tiles and builds a spaceship. Not turn-based, just taking tiles and placing them as fast as you can.
    
2.  We fly our ships through space, with cards telling us what happens. Things like, a meteor storm comes from the left, where a better-built (or luckier) ship is better able to resist them.
    

The first half is silly and fun, and pretty different from other games I've played. The second half takes too long and isn't very interesting, even with the "rough roads" cards. There's an [expansion](https://boardgamegeek.com/boardgame/360232/galaxy-trucker-keep-trucking) that just came out ([rules](https://www.jefftk.com/galaxy-trucker-keep-on-trucking-rules-en.pdf)) but it doesn't look like it fixes this.

Instead, I want a completely new second half. Either something with good momentum that adds depth to the first half, or something simple and short that gets out of the way so you can play more first half.

Here's a non-playtested version of the latter:

*   There are five categories: ship speed, open connectors, crew size, laser count, shield coverage.
    
*   You get one point for coming in first in a category, two for second, etc.
    
*   Category order is randomized and you each decide independently whether to use batteries.
    
*   Fewest points wins.
    

I think this is probably not the right direction to take the game: there are lots of things you could do with these ships. Searching "galaxy trucker alternate ending" doesn't turn anything up, but if someone wanted to put something together I'd be very excited to try it. In the meantime, I think I probably would prefer my boring-but-quick counting version of the second half to the real one.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0LgoXDj7tGDXvzo1z1Uu6mdY9Mm5sFGjNfS8xUXYQ7x3HPJGNHgzLDXUPLoH3NiTXl)*
# Linkpost: Github Copilot productivity experiment

> We recruited 95 professional developers, split them randomly into two groups, and timed how long it took them to write an HTTP server in JavaScript. One group used GitHub Copilot to complete the task, and the other one didn’t. We tried to control as many factors as we could–all developers were already familiar with JavaScript, we gave everyone the same instructions, and we leveraged GitHub Classroom to automatically score submissions for correctness and completeness with a test suite. We’re sharing a behind-the-scenes blog post soon about how we set up our experiment!
> 
> In the experiment, we measured—on average—how successful each group was in completing the task and how long each group took to finish.
> 
> *   The group that used GitHub Copilot had a **higher rate of completing the task** (78%, compared to 70% in the group without Copilot).
> *   The striking difference was that **developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot**. Specifically, the developers using GitHub Copilot took on average 1 hour and 11 minutes to complete the task, while the developers who didn’t use GitHub Copilot took on average 2 hours and 41 minutes.

My opinion: Because of the usual reasons (publication bias, replication crisis, the task being "easy," etc.) I don't think we should take this particularly seriously until much more independent experiments have been run. However, it's worth knowing about at least.   
  
Related: [https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html](https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html)

> We compare the hybrid semantic ML code completion of 10k+ Googlers (over three months across eight programming languages) to a control group and see a 6% reduction in coding iteration time (time between builds and tests) when exposed to single-line ML completion. These results demonstrate that the combination of ML and SEs can improve developer productivity. Currently, 3% of new code (measured in characters) is now generated from accepting ML completion suggestions.
# Should you refrain from having children because of the risk posed by artificial intelligence?

Eli Lifland discusses AI risk probabilities [here](https://www.foxy-scout.com/wwotf-review/).

Scott Alexander talks about how everything will change completely in [this post](https://astralcodexten.substack.com/p/slightly-against-underpopulation), and then [says](https://astralcodexten.substack.com/p/slightly-against-underpopulation/comment/8158683) "There's some chance I'm wrong about a singularity, there's some chance we make it through the singularity, and if I'm wrong about both those things I'd rather give my kid 30 years of life than none at all. Nobody gets more than about 100 anyway and 30 and 100 aren't that different in the grand scheme of things. I'd feel an obligation not to bring kids into a world that would have too much suffering but I think if we die from technological singularity it will be pretty quick. I don't plan on committing suicide to escape and I don't see why I should be not bringing life into the world either.". I have never seen any convincing argument why "if we die from technological singularity it will" have to "be pretty quick".

Will MacAskill [says that](https://80000hours.org/podcast/episodes/will-macaskill-what-we-owe-the-future/#transcript) "conditional on misaligned takeover, I think like 50/50 chance that involves literally killing human beings, rather than just disempowering them", but "just" being disempowered does not seem like a great alternative, and I do not know why the AI would care for disempowered humans in a good way. 

It seems to me that the world into which children are born today has a high likelihood of being really bad. Is it still a good idea to have children, taking their perspective into account and not just treating them as fulfilling the somehow hard-wired preferences of the parents?

I am currently not only confused, but quite gloomy, and would be grateful for your opinions. Optimistic ones are welcome, but being realistic is more important.
# Interpreting Affordable Housing

In describing affordable housing you'll often see a project described as "60% AMI", which can be confusing:

> *how is 60% of median income affordable?*—[NUMTOT comment](https://www.facebook.com/groups/whatwouldjanejacobsdo/posts/2248919421934333/?comment_id=2249414685218140)

If you think that "60% AMI" housing charges 60% of the area median income, I agree that doesn't sound affordable! The problem is, "affordable housing" (and "60% of area median income") is an [improper noun](https://siderea.dreamwidth.org/1773806.html), a technical term with a defined meaning that differs from how we normally use the words.

When housing is described as "affordable", that means it costs 30% of some specific income. If it's a "60% AMI" unit, that means it's affordable to someone making 60% of the area median income. For example, a 60% AMI unit in an area with a median income of $100k would be 60% * 30% * $100k/y or $18k/y, which is $1500/month. A 30% AMI unit would be half that, at $750/month. If you don't know about the factor of 30% thing you end up thinking all this housing is ~3x more expensive!

That said, this is a mildly silly way to do it: percentages of a median rarely make sense, and the right way to do this is probably with percentiles. Instead of targeting affordable housing at, say, the 60% of median income, we should target a 30th-percentile income. This is much more robust to changes in the shape of the income distribution, especially in cases where areas get into some sort of bimodal situation where a bit more than half the residents are making a lot and a smaller but significant number are making much less.
# Swap and Scale

*Produced as part of the SERI MATS program under John Wentworth*  
*Thanks to Garrett Barker for pointing out you could think of Swap Symmetry as relabeling and to Matthias G. Mayer for catching my mistake in not multiplying the bias in Scale Symmetry.*

  
Here are two ways you can change the weights of any network using ReLU's without changing the behaviour of the network. The first is a discrete symmetry  and involves swapping nodes around, the second is a continuous symmetry and involves scaling the weights.  (Astute readers will notice that this is where the title has come from.)

I’m currently investigating the behaviour of *non-trivial *symmetries in networks but that is going to be a much longer post and probably a month or two away. I thought describing the trivial ones would make for a fun bite-sized post to write on a Friday afternoon. 

Trivial Symmetries
==================

There are two types of symmetries that occur for any point in parameter space. I refer to these as the trivial symmetries. I’m unsure who the first person to point these out for ReLU neural networks was, but equivalent symmetries for networks using tanh activation functions were mentioned in Bishop’s 2006  “Pattern Recognition and Machine Learning” and cites “Functionally Equivalent Feedforward Neural Networks” (Kurkova and Kainen, 1994) as proving the case for networks in general. In terms of Machine Learning literature this is ancient, and I suspect the authors haven't even heard of Lesswrong. 

Swap Symmetry
-------------

For any hidden layer, we can imagine permuting the weights and biases in such a way that the output to the next layer remains unchanged. Specifically, we apply one permutation to the weights in the “incoming weights” and the biases of the current layer, then apply the inverse permutation to the weights in the next layer. This has the effect of completely undoing the first permutation.  
  
In the simplest example where our layer only has two nodes, we can imagine swapping two neurons around whilst retaining their original connections. 

![](https://lh3.googleusercontent.com/jVQS2Sc3RCwisXijNTk9gIT-4m0ERIXQl1CAuecs9N5tHZWiC7JFzM7NNdms2v88i6HrhuM-eC7NYTB7sSIwnQZ0uwUrAUtrsJg3SFCGB_q6wqeRN4ReEGwKJN-z0u0rVFdrIGm35uW9rqXjTq4dy0a_ATdKZl-ajL3zezlmUlv5N5W8qa9JmfqtoA)

It may also be helpful to think of this symmetry as simply relabelling the nodes in a single layer. For every hidden layer, there are N! such symmetries, where N is the number of nodes in the layer. 

Scaling Symmetry
================

The second symmetry occurs when we multiply the weights leading into a node by some non-zero real number α, multiply the bias by α and multiply the weights leading out of the node by 1 over α. This has the effect of leaving the final weight unchanged, due to the linear nature of the ReLU function. 

  
 

![](https://lh5.googleusercontent.com/dz751eXOKOIjWTLgiD6j5UAJzgx1xat-E6Gl2dZ7DDyN-5O-s9SGe91OC-0TiA84EN6KhVzdMBc_nKV0AF7XzV5xfm2zvDDIpGoi4CebzWkC3lXP9fR1VtKYDsjRhU1y2uOyX91WZzFWTawl4hrTqKxpBw3I6Lk_s3U2oEskAT2rLYw-5FIzISBVFA)
# Web4/Heaven - The Simulation

*[This is a thought experiment / ranty idea. I don’t necessarily agree with everything I say hear but try to think the idea through to its end. Inspired by the hypothetical practice of heaven banning, hence why I call it Heaven.]*

Web4 is the internet that is completely (or at least mostly) replaced by AI generated content. Any content you consume or interactions you have are replaced by AI: GPT-30 generates responses to you comments on Web4-Reddit and posts you favorite tweets on Web4-Twitter. Dalle-10 generates any image you might choose to look at in the internet and VideoDalle creates your favorite Web4-YouTube videos. Every internet user has their own version of the internet, completely tailored to them, perfectly adjusted to maximize everyone’s enjoyment of time spent on the internet, learned from the users’ individual feedback. Why would you need real Reddit posters or Instagram influencers, when Web4 Instagram contains only images that are perfectly generated specifically you? 

You might say, posting a comment under a YouTube can provide helpful feedback to the content creator, but your comments under Web4-YouTube videos can also influence what videos VideoDalle will show you in the future.

Some people think we already live in a simulation — this is that but for the internet. Even if you know that your whole online experience is simulated, how many of your online interactions ever make it into the real world? Does it matter whether there are actually real people behind those?

Sure, there is still space to get posts from your real life friends on Web4-Facebook. Maybe you even choose to have generated content be interspersed with real people, this way you can still make real life friends on the web. Maybe fake YouTube influencers will tell you about real-world news stories, courtesy of MultiModalGPT-20.

No more negative interactions on the internet (unless you want some). Presumably content could even be optimized to fulfill some kind of specific goal you set: Learning, intellectual growth, limiting time spent on the internet.

Having access to the entire internet and sufficiently capable content generation AI systems should be sufficient to create a simulated version of it.

Is this a dystopian vision? Do you take the red pill or the blue pill?
# Ought will host a factored cognition “Lab Meeting”

Ought will host a factored cognition “Lab Meeting” on **Friday September 16 from 9:30AM - 10:30AM PT**. 

We'll share the progress we've made using language models to decompose reasoning tasks into subtasks that are easier to perform and evaluate. This is part of our work on  [supervising process, not outcomes.](https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes) It’s easier for us to show you than to tell you about it in a post (though written updates will hopefully follow). 

Then, we'll cover outstanding research directions we see and plan to work on, many almost shovel-ready. If the alignment community can parallelize this work across different alignment research teams, we can make progress faster. We'd love to coordinate with other alignment researchers thinking about task decomposition, process supervision, factored cognition, and IDA-like approaches (where efficient to do so). We want to save you time and mistakes if we can!

**What is the agenda?**
-----------------------

1.  30 min | Updates on Ought’s work decomposing reasoning tasks
    1.  The specific alignment problems we’re trying to solve and our vision of a solution that mitigates these risks (see also: [Supervise Process, not Outcomes](https://www.lesswrong.com/posts/pYcFPMBtQveAjcSfH/supervise-process-not-outcomes))
    2.  Early progress on decomposing reasoning about evidence quality in randomized controlled trials
    3.  Our tools for building and debugging reasoning traces of language models ([preview](https://twitter.com/stuhlmueller/status/1558159844330262528))
2.  15 min | Related research directions we’re excited about and how they fit in, e.g.
    1.  Automating evaluation through critique models or verifier models
    2.  Distillation
    3.  Comparing scaling trends for process-based systems vs. end-to-end systems
    4.  Testing process-based systems for adversarial robustness
3.  15 min | Q&A

There will be more to discuss than we can fit into an hour. We’ll get to what we can and consider making this a regular meeting if there’s appetite (likely with more sharing from other researchers)! 

**Who should attend?**
----------------------

You should attend if:

1.  You are interested in Ought’s research and want updates. 
2.  You want to build off of Ought’s learnings from doing this research.
3.  You want to use our tools for running and debugging compositional language models tasks.
4.  You want concrete research ideas in this domain. 
5.  You are not a researcher but want to learn how other backgrounds can support this work (engineers can build debugging infrastructure, non-ML researchers can help create datasets, etc.). 

**How can I attend? **
----------------------

You can register for the Lab Meeting [here](https://us02web.zoom.us/meeting/register/tZMrduCgrTgiEtaVJvxKgMMs_QOHSFznq2Dx). Email [jungwon@ought.org](mailto:jungwon@ought.org) if you have any questions!

The meeting will be recorded & shared.
# Keeping Time in Epoch Seconds

Unix-like computers keep time in seconds since the [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time)—the 0 second, which is set to midnight, January 1st, 1970, Zulu/UTC. We call a particular second since the Unix Epoch a timestamp.

Keeping time this way is pretty handy, but looking at a long string of numbers is hard to parse. What to do?

Right now, as I'm typing, the current Unix timestamp is `16'62'7'68'531`. See how I put those `'` in there? That's how I try to break up the number into parts to make sense of it.

Starting from the right, the two rightmost digits make up what I think of roughly as a Unix minute, but rather than 60 seconds it's 100, so that's $1 \frac{2}{3}$ standard minutes. One more unit over and we get the Unix hour, which is about a quarter of a standard hour long at 1000 seconds or $16\frac{2}{3}$ standard minutes.

Calling 1000 seconds an hour might seem confusing, but the purpose of hours is to have a convenient unit for dividing up days, just like minutes are a convenient way of dividing hours. Speaking of which, 100 Unix hours equates to 1 Unix day, or $27 \frac{7}{9}$ standard hours. This is a bit longer than a standard day, but since some people seem to operate on [28 hour days](https://www.explainxkcd.com/wiki/index.php/320:_28-Hour_Day) anyway, they might find this appealing.

So that takes care of the 5 rightmost digits: 3 to track the seconds and minutes within an hour and 2 to track the hour within a day. The 6th digit from the right tracks the day of the Unix week, which is 10 Unix days, or $11\frac{31}{54}$ standard days. It's a bit longer than our standard 7 day lunar weeks, but 10 day weeks were good enough for the ancient Greeks, so they're good enough for Unix nerds.

Since months are tied to the cycle of the moon there are no real months in Unix time. Instead 100 weeks make up a Unix year, which is approximately 3.17 standard years long. If it helps, though, a tenth of a Unix year is about three and a half months long, so you can think of each ten-week as like the three month quarters or seasons we informally divide our calendar by.

Now we've accounted for all the digits. If we break down my example timestamp of `16'62'7'68'531`, we can say that it was the 31st second of the 5th minute of the 68th hour of the 7th day of the 62nd week of the 16th year of the Unix Epoch. Breaking it down in a table:

| `16` | `62` | `7` | `68` | `531` |
| --- | --- | --- | --- | --- |
| year | week | day | hour | minute and seconds |

Do I expect this to catch on? Not really. This is all just a bit of fun to make sense of Unix timestamps. I can't recall seeing anyone else try to break down the Unix timestamp like this into human-sized units—at best I've seen people talk about things like megaseconds, and I think I can recall others referring to 100,000,000 seconds as a Unix year—so I figured I'd share it. I've been using this in my terminal for years to help me make sense of what I'm looking at. Maybe you'll also find it useful.
# Put Dirty Dishes in the Dishwasher

> *"It's not worth using the dishwasher: by the time you've gotten your dishes clean enough to put in, you might as well just finish washing them yourself."*

People have a wide range of ways they deal with dishes before putting them in the dishwasher:

1.  Rinse well. The dishes should look clean before they go in.
    
2.  Rinse lightly. It's fine if the dishes don't look clean, but they should be close.
    
3.  Scrape well. There shouldn't be anything on the dishes that further scraping would remove.
    
4.  Scrape lightly. Get most of the food off, but it's fine if some stuff remains.
    
5.  Do nothing. Just dump the whole thing in there, food and all. *\[Not recommended.\]*
    

Earlier options are more work, give you cleaner dishes, and mean you need to manually clean the dishwasher's filter less often. Later are the reverse.

In my experience, people tend to be too far towards the early end for their particular dishwasher and time-cleanliness preferences. This makes sense: if you're too far toward the later end you notice and adjust, while in the other direction there's no signal that you could be doing better. Try a later option; you could be putting effort into pre-cleaning for minimal impact on your dishes. It's a cheap experiment to run: there's a good chance you're happy, and if not you can just re-run this load and go back to more thorough pre-cleaning.

Personally, I'm pretty happy with #4. It's very little work, and usually all the dishes come out clean. When one is still dirty, I just include it in the next load after scraping a bit. If dishes come out unusually dirty it's generally because something was loaded wrong (too tall on the bottom rack, poking through the middle rack) and prevented the middle arm from spinning.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02Y3fgWuTMCkwSpvxk7kUTpn2AxPbAcvHWUmVvyPU2Z9GhnoPiGTz95ErzjWCrfsDxl)*
# Making a New Table Leaf

We got our dining room table used, and it didn't come with any leaves. I made one out of plywood when we first got it, and it's been pretty helpful over the past ~7y:

[![](https://www.jefftk.com/old-table-leaf.jpg)](https://www.jefftk.com/old-table-leaf-big.jpg)

With the leaf we can comfortably fit eight: three on the long sides and one on the short. If we have more people two on the short works, but when you get above ten people it's pretty squishy. With our [new housemate](https://www.jefftk.com/p/room-opening), starting to have dinner guests again, and Nora getting close to sitting at the table with us, I wanted to make the table bigger again.

The telescoping mechanism opens to 36", and the current leaf was half that. I considered making a new leaf the same size as the previous, but decided to make a full-size one instead since it would be sturdier. It should also look a bit better, since it will only fail to match the surrounding table and not additionally the earlier leaf.

I cut some 3/4" plywood to the table width and rounded off the corners. I needed to drill four holes for pegs that the table uses to keep the leaf aligned. Here's the initial fit:

[![](https://www.jefftk.com/new-leaf-pre-finishing.jpg)](https://www.jefftk.com/new-leaf-pre-finishing-big.jpg)

I sanded it with 80/120/150/220 grit on the [random orbit sander](https://en.wikipedia.org/wiki/Random_orbital_sander) until it was very smooth, almost soft to the touch. I used Minwax wood finish ("Early American 230"), left over from an earlier project, to stain it. I did two rounds, because last time it ended up lighter than the rest of the table. If you want a better match you can test out a bunch of different things in parallel on the back of the piece or a scrap of identical wood, but that didn't feel worth it here: I'm not even using the same type of wood as the rest of the table. Then three coats of oil-based polyurethane:

[![](https://www.jefftk.com/new-leaf-three-coats.jpg)](https://www.jefftk.com/new-leaf-three-coats-big.jpg)

It was still slightly rough so I sanded it again with 220:

[![](https://www.jefftk.com/new-leaf-three-coats-sanded.jpg)](https://www.jefftk.com/new-leaf-three-coats-sanded-big.jpg)

Another coat of poly, and here's the finished leaf:

[![](https://www.jefftk.com/new-table-leaf-finished-horizontal.jpg)](https://www.jefftk.com/new-table-leaf-finished-horizontal-big.jpg)

[![](https://www.jefftk.com/new-table-leaf-finished-vertical.jpg)](https://www.jefftk.com/new-table-leaf-finished-vertical-big.jpg)

It will get less shiny with a bit of use, and fit in better. You can see that the outer edge is a simple rounded corner, and not the more complex routed design on the original. I'd considered trying to match it, but it would be a lot of work and I don't think most people sitting at the table will care.

Now it's back outside to air out (smells like polyurethane) until we need a big table!
# You Don't Have To Click The Links

[There](https://www.lesswrong.com/posts/46qnWRSR7L2eyNbMA/the-lens-that-sees-its-flaws) [are](https://www.lesswrong.com/posts/3nZMgRTfFEfHp34Gb/the-meditation-on-curiosity) [a](https://www.lesswrong.com/posts/nYkMLFpx77Rz3uo9c/belief-as-attire) [lot](https://www.lesswrong.com/posts/CPP2uLcaywEokFKQG/toolbox-thinking-and-law-thinking) [of](https://www.lesswrong.com/posts/xTyuQ3cgsPjifr7oj/faster-than-science) [blog](https://www.lesswrong.com/posts/5bJyRMZzwMov5u3hW/science-doesn-t-trust-your-rationality) [posts](https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-defense-not-even-science) [that](https://www.lesswrong.com/posts/Mc6QcrsbH5NRXbCRX/dissolving-the-question) [have](https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect) [>5](https://www.lesswrong.com/posts/pGvyqAQw6yqTjpKf4/the-gift-we-give-to-tomorrow) [links](https://www.lesswrong.com/posts/9kcTNWopvXFncXgPy/intellectual-hipsters-and-meta-contrarianism) [to](https://www.lesswrong.com/posts/2brqzQWfmNx5Agdrx/the-virtue-of-silence) [other](https://www.lesswrong.com/posts/895quRDaK6gR2rM82/diseased-thinking-dissolving-questions-about-disease) [blog](https://www.lesswrong.com/posts/yCWPkLi8wJvewPbEp/the-noncentral-fallacy-the-worst-argument-in-the-world) [posts](https://www.lesswrong.com/posts/4xKeNKFXFB458f5N8/ethnic-tension-and-meaningless-arguments). I used to click on all of these, and upon seeing that I hadn't read them, I would embark on a several hour long journey of reading until exhaustion, often times without even finishing the original post I started.

Recently, I had the simple realization that I could usually read a blog straight through without ever reading one of the linked posts while still understanding the content of the blog.

I wanted to share this with other lurkers in hopes that I could make reading on LessWrong more enjoyable and efficient.

Exercise for the reader: don't click the links!
# Why do People Think Intelligence Will be "Easy"?

In many discussions I've hard with people around AI takeoff, I've frequently encountered a belief that intelligence is going to be "easy".   
  
To quantify what I mean by "easy", many people seem to expect that the *marginal* returns on cognitive investment (whether via investment of computational resources, human intellectual labour, cognitive reinvestment or general economic resources) will not be diminishing, or will diminish gracefully (i.e. at a sublinear rate).

(Specifically marginal returns around and immediately beyond the human cognitive capability frontier.)

I find this a bit baffling/absurd honestly. My default intuitions lean towards marginal returns to cognitive reinvestment diminishing at a superlinear rate, and my arm chair philosophising so far (thought experiments and general thinking around the issue) seem to support this intuition. 

A couple intuition pumps:

*   50%, 75%, 87.5%, 93.75%, ... are linear jumps in predictive accuracy (one bit each), but returns seem to diminish at an exponential rate
    *   On the other hand 6.25%, 12.5%, 25%, 50% represent the same linear jumps, but this time with returns growing at an exponential returns
    *   This suggests that the nature of returns to cognitive investment might exhibit differing behaviour depending on where in the cognitive capabilities curve you are
        *   Though I've not yet thought about how this behaviour generalises to other aspects of cognition separate from predictive accuracy
*   Apriori, I'd expect that given how much human dominance is almost entirely dependent on our (collective) intelligence, our evolution would have selected strongly for intelligence until it met diminishing returns to higher intelligence
    *   That is, we should expect that evolution made us as smart as was beneficial given its constraints:
        *   Calorie requirements of larger brains
        *   Energy budget
        *   Size of the human birth canal
        *   Slight variations on existing cognitive architecture
        *   Etc.
    *   [I think the reported higher incidence of some neuronal issues/disorders among higher IQ human subpopulations (e.g. Ashkenazi Jews)](https://slatestarcodex.com/2017/05/26/the-atomic-bomb-considered-as-hungarian-high-school-science-fair-project/#:~:text=Doctors%20have%20long%20noted%20that%20Ashkenazi%20Jews%20are%20uniquely%20susceptible%20to%20various%20genetic%20diseases) may be an argument in favour of evolution having met said marginal returns
    *   Perhaps [the Flynn Effect](https://en.wikipedia.org/wiki/Flynn_effect) is an argument against this

Another way to frame my question this is that there will obviously be diminishing marginal returns (perhaps superlinearly diminishing) at some point; why are we confident that point is far in front of the peak of human intelligence?
# Time is not the bottleneck (on making progress thinking about difficult things)

... at least, not for me. The bottleneck is something like [mental energy](https://en.wikipedia.org/wiki/Mental_energy). I can only make progress on a (subjectively) difficult topic for a handful of hours (at best) each day before hitting **severe** diminishing returns, and being forced to "relax" for the rest of the day. 

In light of the above, the obvious thing to optimize for the sake of "making progress on difficult things" is the amount of "mental energy available per day", not "waking hours available per day". Sleep seems to serve the function of "restoring mental energy"; but I get diminishing returns on sleeping longer (in a single block) as well; for me, there doesn't seem to be a big difference in "available mental energy" during a day whether I'd slept 6 hours or 9 hours.

So, since I'm getting diminishing returns on both my waking and sleeping hours, the obvious thing to try is to split the day up into multiple "sub days", i.e. some form of [polyphasic sleep](https://en.wikipedia.org/wiki/Biphasic_and_polyphasic_sleep). Ideally, I'd like to spend a larger total fraction of the day asleep; this contrasts things like "the Uberman's sleep schedule", where the goal is to have as much waking time as possible (which, through this lens, is pure folly).

Have any of you (or anyone you know) had any success with something like this?
# Which LessWrong content would you like recorded into audio/podcast form?

For the last few months, LessWrong has been working with Solenoid_Entity to record each [curated post](https://www.lesswrong.com/recommendations) into audio form. Recordings to date can be [found here](https://www.buzzsprout.com/2037297/) and on all major Podcast apps when you search "LessWrong".  
  
More recently, we added some UI so it's clear when audio is [available on a post](https://www.lesswrong.com/posts/rYDas2DDGGDRc8gGB/unifying-bargaining-notions-1-2).  
 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/623619317705980d2c60cea6362bd785f42eb7c56386858d.png)

click the green speaker icon to display audio UI when available

We are now scaling up the amount of audio recorded, and the question is, what should we prioritize?

We can records posts, Sequences, wiki-tags, etc...
# Simple proofs of the age of the universe (or other things)

I saw some time ago Carl Sagan asked schoolchildren for proofs that the earth is round. I can no longer find it, but they gave him sound, simple answers. One was watching a ship seem to sink below the horizon as it grows distant. One was photos of Earth from space. One, I believe, was seeing how the sun may reach the bottom of a well in the tropics, but not further north.  
  
I'm trying to find similar answers related to the age of the universe. I'll operationally define "old" as "older than human history." So far:  
\* We know by parallax distances to various stars within 300 l y, and can tell from their distance, color, and brightness how bright a star of a given color should be. So if we see a further star and measure its color and brightness, we can deduce the distance and thereby how long it took the light to get here. (This is more complicated than I'd like, but oh well.)  
\* If the universe started X thousand years ago, wouldn't we be seeing more stars appearing every year at the X-thousand l y range?   
\* There is a star that exploded? did a nova? and some decades later its light illuminated a nearby nebula. By the time it took to illuminate, we know the distance between star and nebula. Since we know the angle, we can get the length of the other legs of the triangle, which are further than light could travel in human history. (I can't remember the name of the star, alas.)  
\* Carbon dating.  
\* The Grand Canyon's sedimentary layers.  
\* South America and Africa look like they fit together. If they traveled at current speeds of continental drift, they couldn't have reached their current distance in a few thousand years (plus, their animal populations would have mixed when they were touching).

What else?

It's interesting to think of how this works on other questions. Germs, or spontaneous generation of life? proved by canned food. Oxygen, or phlogiston? proved by a sealed-up candle. Quarks?
# VA Rationalists - Norfolk Meetup

Hi!

Come on out to Norfolk's weekly social meetup, all are welcome :)

Suggested reading: [Twelve Virtues of Rationality](https://www.lesswrong.com/posts/7ZqGiPHTpiDMwqMN2/twelve-virtues-of-rationality)

We meet on Wednesdays from 17:00-19:30 EDT.

Cheers,  
Willa
# Simple 5x5 Go

I've been [playing 5x5 Go](https://www.jefftk.com/p/5x5-go-board) with a lot of people new to the game, and have been using some [simplified](https://www.jefftk.com/p/simplifying-board-games) rules for faster teaching:

1.  You can play on any empty intersection, unless it would make the board [look like it did previously](https://senseis.xmp.net/?PositionalSuperko).
    
2.  If you take away the last liberty ("line leading away") from a group of pieces, they're lifted, opponent's ones before your own.
    
3.  The winner is the only one with pieces left on the board.
    

I'll start them with a number of their pieces already on the board ("handicaps") that I'm guessing will be about balanced. Over time they'll need fewer and fewer.

After playing a bit, often in the first game if they're picking things up quickly, we introduce the idea that if it's clear you're not going to win you resign.

At some point you likely get to a game where both have pieces that can't be lifted. After declaring that a tie, I'll explain territory scoring and passing. If they're the kind of mathy person I think will enjoy it I'll show them the [Tromp-Taylor](http://tromp.github.io/go.html) version, and with that we're ready to move on to larger boards.
# Transhumanism, genetic engineering, and the biological basis of intelligence.

I recently had a [fascinating conversation with Dr. Steven Hsu](https://www.youtube.com/watch?v=Ks8lLlogbOY), theoretical physicist and serial entrepreneur, about computational genomics and genetic engineering.

There were a number of really interesting takeaways which I think would be of interest to LWians.

*   Apparently, genetic engineering in the form of embryo selection is *very* far along. They can already select for things like eye color and propensity for diseases. The only reason they're not selecting for intelligence yet is because there is such a strong stigma and because there's not currently enough labeled training data for the algorithms.
*   The algorithms involved are simpler than you might think, it mostly appears to be basic regression models regularized with LI penalties (as those handle sparse data much better)
*   Dr. Hsu *does* think it's worth worrying about this technology's potential to worsen inequality, and we kick the ethics of that around for a little bit. 
*   Epigenetics plays less of a role in determining life outcomes than many of us have been led to believe. 
*   We also discuss whether AGI or genetic engineering will lead to the first superhuman intelligence.

Hope you enjoy it!
# Precise P(doom) isn't very important for prioritization or strategy

People spend time trying to determine the probability that AI will become an existential risk, sometimes referred to as P(doom).

One point that I think gets missed in this discussion is that a precise estimate of P(doom) isn't that important for prioritization or strategy.

I think it's plausible that P(doom) is greater than 1%. For prioritization, even a 1% chance of existential catastrophe from AI this century would be sufficient to make AI the most important existential risk. The probability of existential catastrophe from [nuclear war](https://www.lesswrong.com/posts/sT6NxFxso6Z9xjS7o/nuclear-war-is-unlikely-to-cause-human-extinction), pandemics, and [other catastrophes](https://forum.effectivealtruism.org/posts/GsjmufaebreiaivF7/what-is-the-likelihood-that-civilizational-collapse-would) seems lower than 1%. Identifying exactly where P(doom) lies in the 1%-99% range doesn't change priorities much.

Like [AI timelines](https://www.lesswrong.com/posts/d7RFGPMkbKurQjW6o/agi-timelines-are-mostly-not-strategically-relevant-to), its unclear that changing P(doom) would change our strategy towards alignment. Changing P(doom) shouldn't dramatically change which projects we focus on, since we probably need to try as many things as possible, and quickly. I don't think the list of projects or the resources we dedicate to them would change much in the 1% or 99% worlds. Are there any projects that you would robustly exclude from consideration if P(doom) was 1-10% but include if P(doom) was 90-99% (and vice versa)?

I think communicating P(doom) can be useful for other reasons like assessing progress or getting a sense of someone's priors, but it doesn't seem that important overall.
# ACT-1: Transformer for Actions

> ACT-1 can take a high-level user request and execute it. The user simply types a command into the text box and ACT-1 does the rest. In this example, this requires repeatedly taking actions and observations over a long time horizon to fulfill a single goal.

...

> While we’re excited that these systems can transform what people can do on a computer, we clearly see that they have the potential to cause harm if misused or misaligned with user preferences. Our goal is to build a company with large-scale human feedback at the center — models will be evaluated on how well they satisfy user preferences, and we will iteratively evaluate how well this is working as our product becomes more sophisticated and load-bearing.

Daniel's commentary: To be clear, this is very much just a cool demo, not much better than WebGPT as far as I can tell, and not surprising at all that this level of capability is possible.
# Seeing the Schema

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d55881b5728a40d710caddc28a111bf20518925aff8c5167.png)

For the past month or two, I've been regularly playing a mental rotation game, available [here](https://0xf00ff00f.github.io/rotator/). In the game, the task is to find which two of the six displayed block objects are identical under rotation. I'm not the best at mental rotation, so I figured I'd try and get better at it. But while I've gotten better at the matching task, I think I've started to see diminishing returns at mental rotations - primarily because, for the most part, I've stopped doing them.

The "reward signal" here is success at the matching task - which doesn't actually require you to perform a mental rotation. In the above image, for example, I run a few basic heuristics: the two blocks on the right are planar, and as one of them has a hook shape and the other doesn't, they can't match. Then, the lower centre block doesn't have a planar hook shape, where the other three do, so it can't match any of them. Finally, of the remaining blocks, the top centre's extruded "pole" above the planar hook is two blocks high, where the two on the left have extrusions three blocks high. As such, the two on the left must match.

It's easier and faster for my brain to run these "checksums" than it is to actually perform the mental rotation task. Early on, I *was* actually performing mental rotation, but now I'm just running heuristics across the set of shapes. This might seem like a pathological or degenerate solution - I'm not actually training the "thing I wanted" any more - but it's the path that gives me the highest reward at the end, so that's the thing my subconscious brain's started defaulting to.

It's weird to see a misaligned reward signal from the inside.
# Forecasting thread: How does AI risk level vary based on timelines?

*Crossposted to* [*EA Forum*](https://forum.effectivealtruism.org/posts/nYgw4FNpHf9bmJGEi/forecasting-thread-how-does-ai-risk-level-vary-based-on)

While there have been many previous surveys asking about the chance of existential catastrophe from AI and/or AI timelines, none as far as I'm aware have asked about how the level of AI risk varies based on timelines. But this seems like an extremely important parameter for understanding the nature of AI risk and prioritizing between interventions.

Contribute your forecasts below. I'll write up my forecast rationales in an answer and encourage others to do the same.

Elicit Prediction ([forecast.elicit.org/binary/questions/726u\_r\_XC](forecast.elicit.org/binary/questions/726u_r_XC))

Elicit Prediction ([forecast.elicit.org/binary/questions/8HnkL7Ekf](forecast.elicit.org/binary/questions/8HnkL7Ekf))

Elicit Prediction ([forecast.elicit.org/binary/questions/dlPDzREFM](forecast.elicit.org/binary/questions/dlPDzREFM))

Elicit Prediction ([forecast.elicit.org/binary/questions/EHL2INyQq](forecast.elicit.org/binary/questions/EHL2INyQq))

Elicit Prediction ([forecast.elicit.org/binary/questions/AETDAUmNF](forecast.elicit.org/binary/questions/AETDAUmNF))
# What's the longest a sentient observer could survive in the Dark Era?

Periodically I look at [this graph](https://en.wikipedia.org/wiki/Future_of_an_expanding_universe), and I'm like, "holy shit, that is *so much Nothing.* Wut. Like, they converted it to log-scale so it would fit into my brain and it's still *so much Nothing, Jezus Christ.*"

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/40dedc69fef937deb2ee8e8148c785d4383acefcec8bf4b2.png)

Eliezer's Dath Ilan verse has this song for people who died before the invention of cryonics:

> Even if the stars should die in heaven  
> Our sins can never be undone  
> No single death wil be forgiven  
> When fades at last the last lit sun.  
> Then in the cold and silent black  
> As light and matter end  
> We’ll have ourselves a last look back  
> And toast an absent friend.

And I find myself wondering "What are the physical limits on 'some sentient computation surviving into the dark era?" Assuming we don't learn anything new about physics, or acausal-trade our way into another universe or whatever, what's the farthest to the right on this timeline you could possibly get, if you goal was to have some computation that, as late as possible, looks around at the universe one last time, comprehends it, and says "yep, still full of infinite blackness. Goodbye, world."

Vaguely relevant things I'm vaguely aware of include:

1.  Something something you can somehow harvest energy from black holes. Does this plausibly get us to the end of the Black Hole Era or is that energy eventually too weaksauce?
2.  Space is, like, really cold (especially when All Has Become Darkness), so any time-capsule robot would face an uphill battle not radiating it's energy into the void.
3.  On long enough timescales, protons (probably?) decay, and... like, your robot would eventually... just... disintegrate?

If anyone knows enough physics to satisfy my random curiosity, uh, I'd appreciate it.