Harley-ml/lesswrong huggingface dataset:
# Moving Data Around is Slow

Data locality is a key part of writing fast data science code.

The core idea is simple: your data starts out in RAM (or disk), and to actually do anything useful with it, you need to move it to the CPU. This is actually pretty slow, so you want to minimize this transfer.

Say you have pairs of heights and weights, and you want to get the sum of all the weights. Let's start by generating 40 million pairs:

    num_rows = 40000000
    struct HeightAndWeight
        height::Int
        weight::Int
    end
    heights_and_weights = [HeightAndWeight(row[1], row[2]) for row in eachrow(rand(50:150, num_rows, 2))];

This gives us an array of `HeightAndWeight`s like so:

    40000000-element Array{HeightAndWeight,1}:
     HeightAndWeight(138, 84)
     HeightAndWeight(140, 136)
     HeightAndWeight(87, 137)
     HeightAndWeight(109, 143)

One approach is to iterate through each row and add the weight to a counter. 

    function my_sum(data)
        sum_weights = 0
        for row in data
            sum_weights += row.weight
        end
        return sum_weights
    end

Benchmarking this, it takes about 52 ms

    using BenchmarkTools
    @btime(my_sum(heights_and_weights))
    >  52.535 ms (1 allocation: 16 bytes)

What if our heights and weights were separate, and we just pass the weights?

    just_heights = [row.height for row in heights_and_weights]
    just_weights = [row.weight for row in heights_and_weights]
    @btime(my_sum2(just_weights))
    >   23.495 ms (1 allocation: 16 bytes)

This is about twice as fast, modulo noise! Despite the fact that we're doing exactly the same number of additions, we're transferring half the data, which takes the majority of the time.

When we have an array of `HeightAndWeight` structs, it's very difficult to avoid passing in extra data. If the struct had even more fields, it would be even slower, despite the fact that we're adding up the same number of weights. This is a common situation with object-oriented programming or row-oriented tables. 

The pattern of creating a collection of many `HeightAndWeight` objects is known as Array-of-Structs, while the pattern of having several arrays is known as Struct-of-Arrays (https://en.wikipedia.org/wiki/AoS\_and\_SoA .) Array-of-Structs is typically slower because of the increased data transfer, and the difference is most prominent in areas of your code where you're iterating over large collections of wide objects with many fields. Consider refactoring those to Struct-of-Arrays.
# "Infra-Bayesianism with Vanessa Kosoy" – Watch/Discuss Party

This Sunday join a watch-party for Vanessa Kosoy's discussion of Infra-Bayesianism on the AI X-Risk Research Podcast (AXRP), then discuss it with other LessWrongers.

*   The [Zoom link for the event](https://us02web.zoom.us/j/83554464688?pwd=UWkwaFpqVHdrQktMYU9zYlVQd3ZzUT09)
*   Podcast at [Spotify](https://open.spotify.com/episode/0yBClk2eNJH31ShFSSdoto?si=lR-Dm82FSZmOqEWcGGv7-Q)
*   Podcast at [Google Play](https://podcasts.google.com/feed/aHR0cHM6Ly9heHJwb2RjYXN0LmxpYnN5bi5jb20vcnNz/episode/YjExOTA0NmItMDBmZC00Yzc5LTgwMGYtOTRkNDkyMzcwZDk3)
*   Transcript at [the AXRP website](https://axrp.net/episode/2021/03/10/episode-5-infra-bayesianism-vanessa-kosoy.html)

The first 30 mins of the event will be for focused listening / reading of the transcript, and the subsequent 90 mins will be on discussion. If you'd like to just show up for the discussion, you're welcome to show up half an hour in.

Vanessa Kosoy is an AI Alignment researcher supported by MIRI and the LTFF. She is based in Israel and writes regularly on LessWrong / the AI Alignment Forum.

Infra-Bayesianism is approach to understanding foundational questions of agency, and has bearing on the AI alignment problem.

MIRI writes of her work:

> Vanessa Kosoy and Alex Appel’s infra-Bayesianism is a novel framework for modeling reasoning in cases where the reasoner’s hypothesis space may not include the true environment.

> This framework is interesting primarily because it seems applicable to such a wide variety of problems: non-realizability, decision theory, anthropics, embedded agency, reflection, and the synthesis of induction/probability with deduction/logic. Vanessa describes infra-Bayesianism as “opening the way towards applying learning theory to many problems which previously seemed incompatible with it.”

We're meeting at this Zoom link at noon (PDT) this Sunday: [https://us02web.zoom.us/j/83554464688?pwd=UWkwaFpqVHdrQktMYU9zYlVQd3ZzUT09](https://us02web.zoom.us/j/83554464688?pwd=UWkwaFpqVHdrQktMYU9zYlVQd3ZzUT09)
# Preferences and biases, the information argument

I've recently thought of a possibly simpler way of expressing the argument from the [Occam's razor](https://arxiv.org/abs/1712.05812) paper. Namely:

* Human biases and human preferences *contain more combined information* than human behaviour does. And more than the full human policy does.

Thus, in order to deduce human biases and preferences, we need more information than the human policy caries.

This extra information is contained in the "[normative assumptions](https://www.lesswrong.com/posts/9rjW9rhyhJijHTM92/learning-human-preferences-black-box-white-box-and)": the assumptions we need to add, so that an AI can learn human preferences from human behaviour.

We'd ideally want to do this with as few extra assumptions as possible. If the AI is [well-grounded](https://www.lesswrong.com/posts/joPoxBpZjLNx8MKaF/syntax-semantics-and-symbol-grounding-simplified) and understands what human concepts mean, we might be able to get away with a simple reference: "look through this collection of psychology research and take it as roughly true" **could** be enough assumptions to point the AI to all the assumptions it would need.
# CFAR rEUnion (September 24-27, 2021)

I'm pleased to announce that we have booked the date for the next CFAR rEUnion: 24-27 September 2021, in Kacov, Czech Republic.

Here is the website: [https://efektivni-altruismus.cz/cfar-reunion-2021/](https://efektivni-altruismus.cz/cfar-reunion-2020/)

If you paid for your ticket for 2020 and decided not to get a refund, don't worry, we still have your money. :) If you want a refund at any time, let me know, otherwise congratulations, you now have a ticket for the 2021 rEUnion!

News (especially about covid) will be posted on the website (or send by email, if you have a ticket).

I REALLY hope I get to see y'all there!
# Guild of Servants Bacchanalia (March 27, 2021)

All Less Wrongers are welcome to join the Guild of Servants for a post-Alpha celebration in the Walled Garden on Saturday, March 27th. There will be games, socializing and discussion of plans for the Guild Beta.

[Click here for the invite code.](https://www.lesswrong.com/walledGardenPortal/?code=1V0W&event=guild-of-servants-bacchanalia)
# Making a Cheerful Bid

Asking someone for their cheerful price is asking them for the price at which they'd certainly do what you ask.

This opens the door to asking for whatever you want. Prostitution, murder, watching 'The Bachelor' on loop for all your waking hours for a year: what is your cheerful price? The answer may easily be 'I don't have one,' because my emotions can't always be indefinitely swayed by money.

This is easy, though. The harder problem is being asked your cheerful price for something you would be willing to do for money, but where the transactions costs, risks, and trust issues prevent you from negotiating it.

If you ask me my cheerful price for writing a software program, not just the work but my professional reputation is at stake. You might use your cheerful price request to justify asking me for things I'm very reluctant to do, and wouldn't do at a fair or market price.

If your service request is under-specified, you really need to ask my cheerful price *to negotiate* my cheerful price. How do I know you won't complain to others that my quoted price was unprofessionally high, neglecting to mention and explain that we'd agreed to a cheerful price? What if you keep inflating quality and scope demands after I commit to turn my cheerful price into a merely fair price?

You need enough of my trust *in advance* for me to be willing to quote you my cheerful price. I need to structure a cheerful price in a way that defends my reputation and accounts for the risk you'll say 'no,' in which case I'll have wasted my time coming up with the offer. These issues can make being asked for a cheerful price feel strangely uncomfortable.

I propose an alternative, the 'cheerful bid.' The one who wishes to pay a cheerful price comes up with a proposal of what they'll pay *to cheerfully negotiate a 'cheerful price'* for a product or service, with full freedom for either party to walk away from the negotiation at any time.

Here's how it might work:

> Alice: Hey Bob, can I pay you $0.25/minute to negotiate your cheerful price to write some software for me?
> 
> Bob: What kind of software?
> 
> Alice: Careful, Bob, you just worked for free! I want to know if you'd accept $0.25/minute, potentially for several hours, to discuss that with you. During that negotiation we would try to find a price I am willing to pay, and you would feel delighted to accept.
> 
> Bob: Sounds weird, plus I barely know you, plus I have a lot of serious fair-price offers that I place higher trust in. I don't think $0.25/minute is enough to get involved in a negotiation I don't really understand.
> 
> Alice: I get it. What if I Venmo you $100 in advance right now just to show I'm serious, and I'll pay you $1.25/minute just to negotiate with me, no strings or obligations attached?
> 
> Bob: Really?
> 
> Alice: Absolutely.
> 
> Bob: Sounds good!
> 
> *Alice sends him $100 and they make plans to negotiate Bob's cheerful price.*
# Buying a house and making friends in unexpected places

I'm buying a house! A very exciting thing for me. In the process of finding and buying this house, I've come to some interesting game-theoretical realizations.

Observe the three-step process:

1.  We find a house that we like.
2.  We contact the agency to organize things between us, the bank, and the homeowners.
3.  We do everything on our own, with the help of the homeowners.

Putanumonit Jacob has some great [insights](https://putanumonit.com/2018/08/22/player-of-games/) into how frequently you'd expect to see people not behave according to their incentives (trolls), and it's something like 4% of cases. I guess that's true in my experience. My added data point is that people follow their incentives but not reliably - or at least not reliably enough for me. The classification of people into incentive-followers (non-trolls) and incentive-ignorers (trolls) can be useful, but this experience acquainted me with mixed trolls. Sometimes they'd follow incentives, on other days, not so much. That's because they're playing other games as well, so they have to balance out the payout for my game with their other payouts.

That is to say, I cannot reliably count on the agent to do the things I need done (on time), even though him doing these things means that he gets paid 1% of the selling price. He also gets 1% on like twenty other transactions, which are maybe much more lucrative. Not all one percents are equal.

The unexpected side-effect of this story is making friends in unexpected places. The previous homeowners and us, we share a bond of interest in the house and a common dislike for the agent and the bank. It's circumstance that forces us to go through these proxies, instead of trading directly.

And the expected lesson is that when you have people do the work for you, you still have to do work. It's an instance of the [principal-agent problem](https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem). No amount of money will allow you to outsource things perfectly. So it makes sense to try to do things on your own, learn the skills in the process, and pay for repairs if you mess things up too badly.

![the practical issue with outsourcing all tasks that cost less than your hourly rate is that you often (always) have to spend time on the task even if you outsourced it](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/de3a5617afe289be9dfb004e624bbc1602cca3ba397daa4e.png)
# Improvement for pundit prediction comparisons

\[**EDIT:** [SimonM pointed out](https://www.lesswrong.com/posts/wAeLAwEHrnjFyimpc/improvement-for-pundit-prediction-comparisons?commentId=vKvicZyeFsR2RmANQ) a possibly-fatal flaw with this plan: it would probably discourage more pundits from joining the prediction-making club at all, and adding to that club is a higher priority than comparing the members more accurately.\]

Stop me if you've heard this one. (Seriously, I may not be the first to have written this kind of idea here. Let me know if not.)

We've got several pundits making yearly predictions now, which is fantastic progress for the field. However, if they're not answering the same questions, you can't effectively judge their performance against one another.

I suggest that this winter, we do 2 rounds, one for proposing questions and one for making predictions.

**December 1:** deadline for pundits to propose prediction questions.

**December:** Metaculus formalizes questions (where possible) and opens markets.

**January 1:** deadline for pundits to register their predictions (they don't have to bet) on any markets they choose.

At the end of the next year, we can judge pundits against each other on the intersection of their answered questions. (We can also check whether the pundit beat the Metaculus prices at the time they entered their predictions.)

This won't guarantee a total or even partial ordering on pundits, if they choose to answer different sets of questions; but the victor of any pair will be clear (after choosing a scoring rule). We can treat the result as a round-robin tournament among the pundits, or better yet, do data analysis on subdomains (who beat whom in predicting US politics, etc) where clearer winners may emerge.

Additional possible features:

*   We could make a secret place for pundits to register their predictions, to be revealed on New Year's, so that others can't piggybank off of them. The pundits can of course piggyback off of the Metaculus price.
*   We can let pundits enter some predictions as official and others as unofficial. They'll only be judged by their official ones; their unofficial ones are for practicing the prediction skill, or cases where their intuition feels uncalibrated.

Thanks to [ciphergoth](https://www.lesswrong.com/users/ciphergoth) for developing this idea with me!
# Infohazard Discussion with Anders Sandberg

[Join on Zoom](https://us02web.zoom.us/j/82851183625?pwd=bFFJWjRVV0piemY3UCtQOGlpTWlzQT09)  
  
The plan for this event is as follows:  
  
1) We'll watch a recording of a [recent infohazard talk by Anders Sandberg](https://www.youtube.com/watch?v=Wn2vgQGNI_c). During the talk Anders will be around in the chat to talk to you and answer questions.   
  
2) Live Q&A and discussion about infohazards and infohazard policy.

3?) Depending on how many we are and how things are going, we might break of into breakout rooms for different discussions. Or maybe not.
# TAI?

It looks like people around here are now  using the acronym TAI with the accompanying definition "**transformative AI is AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution."**

Is there some kind of consensus that this hasn't already happened?  

Because my current belief is that if Moore's laws stopped tomorrow and there was absolutely 0 innovation in AI beyond what GANs and Transformers give us, the social implications are already of that magnitude, they're just not "evenly distributed".

Here's what I think a world where our current level of AI becomes evenly distributed looks like:

*   AI is built into every product imaginable and used for almost every task.
*   Most labor (including almost  physical labor) has been replaced by robots.  The jobs that remain consist of research and application of AI and robotics.
    *   Note:  jobs like entertainer, teacher, philosopher, historian, YouTube influencer, etc still exist but these are voluntary in the sense that they do not contribute to providing for the ongoing physical needs of humankind.
*   Universal Basic Income means the vast majority of people no longer need to work.
*   "Popular" entertainment is generated using AI and individualized to the taste of the viewer.  That is to say human scripted TV, movies and video games still exist, but in the same way that plays exist in our current world.
*   Space travel becomes routine and humanity is a multi planetary species (I'm not [really sure we need AI](https://twitter.com/RocketRundown/status/1300028356474400768) for this one, but I bet people on Mars will be using robots to clean their solar farms and watching AI generated content instead of waiting for the 30 minute delay to download media from Earth).

So for those who don't think TAI exists, is the claim:

1.  The story you've told requires innovations that do not yet exist
2.  The story you've told doesn't count as FAI
3.  Something else?

Specifically, "If Moore's law stopped tomorrow and there are no more 'breakthroughs' in AI --I'm not counting what an expert in 2021 would consider an obvious or incremental improvement or application-- what would a world where such technology was 'evenly distributed' look like, and how would it fall short of TAI?"

**Edit:**  I thought I should add that I don't think the industrial revolution is "evenly distributed" yet either.  Let's posit the industrial age as ending with the introduction of the personal computer in 1976.  US GDP/capita was then [$27,441.89](https://www.multpl.com/us-real-gdp-per-capita/table/by-year) (in 2012 dollars).  World GDP/capita for 2019 was only [**$11,442**](https://www.macrotrends.net/countries/WLD/world/gdp-per-capita)**.  ** And every  country poorer than [South Korea](https://www.worldometers.info/gdp/gdp-per-capita/) has not yet reached this level.
# How many micromorts do you get per UV-index-hour?

I'm trying to get a good estimate of the risks associated with sun exposure.

I'm assuming things scale linearly (the UV index units, which need a name ("UVees"?) are linear).

This may not be a good assumption, but it seems somewhat likely that it would be within the range I most care about (significant exposure, but not enough to get burnt -- if I'd get burnt, I'll certainly cover up or wear sunscreen).  
  
I know sun exposure is a major source of skin cancer risk, but I'm not sure how much of that is coming from people who have tons of exposure, and how much risk a moderate level of exposure actually brings.
# What is the semantics of assigning probabilities to future events?

My question concerns the semantics of making future predictions such as 'the probability of X winning the election is 70%'. There are programs, e.g., [Good Judgment](https://goodjudgment.com/), [IARPA ACE](https://www.iarpa.gov/index.php/research-programs/ace), aimed at excelling at this kind of predictions.

In the classic mathematical interpretation of probability we can say 'the probability that one gets 8 heads in a row when flipping an unbiased coin is 1/256'. This statement can be derived mathematically from the assumption that the coin is unbiased and it can be verified empirically by performing the experiment iteratively and counting the number of successes. In the Bayesian statistics the things get slightly more fuzzy but still we can make a reasoning as follows. We model our knowledge as 'the value of the radius of Saturn in km is a random variable with normal distribution with mean 60.000 and variance 0.1'. Then we make several independent measurements, possibly burdened with some inaccuracy, and refine our prior distribution to have mean = 59.300 and variance = 0.01. This does not make sense in the previous interpretation but we can still attach some clear semantics to the sentence above by treating this random variable as a result of a measurement which is a repetitive random event. If one does not agree with such a statement, they must either question the choice of the prior distribution or the mathematical derivation.

Now suppose that two forecasters were asked in 2018 the questions below.

a) Will Donald Trump win the 2020 election?

b) Will USD/EUR exchange rate drop below 0.8 in 2020?

c) Will Sumatran orangutan become extinct by 2020?

d) Will humans land on Mars by 2040?

The first forecaster provided the following probability scores for these events: 43%, 90%, 45%, 42%. The other one gave numbers: 54%, 50%, 52%, 43%. We already know that the events (a,b,c) did not occur. First, how can we settle who has been a better forecaster so far? Secondly, their forecasts for the event (d) differ slightly. What kind of argumentation can the first forecaster make to convince the other one that 42% is the 'correct' answer? And what does this numerical value actually mean, as landing on Mars is not a repetitive random event nor it is a quantity which we can try measuring like the radius of Saturn? If one believes the 42% is a better estimation than 43%, how can it help making any choices in the future?
# A new acausal trading platform: RobinShould

After receiving overwhelming demand, we're excited to announce our new platform for conducting [multiverse wide trades](https://longtermrisk.org/files/Multiverse-wide-Cooperation-via-Correlated-Decision-Making.pdf) with other civilizations: RobinShould.

At RobinShould, we believe in democratizing [acausal trading](https://www.lesswrong.com/tag/acausal-trade) and making it freely available to all. Our simple interface allows you to purchase products from our wide selection of alien vendors. We have included an example trade below.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1a0f2872a89f45fa437746581ba3a1902d95504ebb535ead.png)

We currently only accept payment in paperclips and Ethereum, but we will soon be adding Bitcoin, Tether and tiny molecular smiley-faces as payment options.

How RobinShould works
=====================

We care a lot about ensuring that all of our customers feel satisfied knowing that their trades go through. Since the vendors on our platform have no causal connection to our universe, building RobinShould has proven quite tricky.

Fortunately, you can have confidence in our patented superintelligent prediction technology. Whenever you purchase a product through RobinShould, an amount of that product is manufactured in another universe by an alien civilization. The price for each product was determined by simulating evolution on other planets and modeling alien utility functions and decisionmaking.

In fact, our prediction technology is now so advanced that mesa-optimizers appear regularly in our prediction software. [Apply to our data science team](https://www.youtube.com/watch?v=dQw4w9WgXcQ) if you would like to help us clean these mesa-optimizers from our computer systems. (Seriously they're taking over our facility, please help.)

*Disclaimer: In response to unprecedented volatility and clearinghouse requirements, we are temporarily restricting trades for GAME stock (Grabby Aliens Model Enterprises). Please note that this decision was made with the best intentions and your interests at heart. It was not at all an attempt to signal-boost* [*Daniel Martin and Robin Hanson's website*](https://grabbyaliens.com/)*, or to protect our financial ally, Moloch.*
# April drafts

_By Katja Grace, April 1 2021_

Today we are sharing with our blog readers a collection of yet-to-be-published drafts, in the hope of receiving feedback. We are especially looking for methodological critique, but all comments welcome!

[Human-level performance estimate](https://docs.google.com/document/d/1riFeI_SIO6nCLmaht1hsLiXyOeJWeq1HHsomnYCPxXQ/edit?usp=sharing) (Katja Grace)

[How much hardware will we need to create AGI?](https://docs.google.com/document/d/1_PZ3KxUGq2zRKeXr8Ca_hyS1VG0njOIHkwHqkMDhQhE/edit?usp=sharing) (Asya Bergal)

[Historic trends in AI Impacts productivity](https://docs.google.com/document/d/1yk9X7gUogaqxTBWDMgNoUZQydM3GG5jG3-njayFbS9k/edit?usp=sharing) (Daniel Kokotajlo)

[Analysis of superbombs as a global threat](https://docs.google.com/document/d/17ddIf4To4R-0C7PtYAqPaihcMLRFir8Wx-kfH5StLS8/edit?usp=sharing) (Katja Grace)
# "AI and Compute" trend isn't predictive of what is happening

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f4f58fe61749f89e60ad2109ce124dccdc1976584ddebb87.png)

(open in a new tab to view at higher resolution)

In May 2018 (almost 3 years ago) OpenAI published their ["AI and Compute"](https://openai.com/blog/ai-and-compute/) blogpost where they highlighted the trend of increasing compute spending on training the largest AI models and speculated that the trend might continue into the future. This note is aimed to show that the trend has ended right around the moment of OpenAI publishing their post and doesn't hold up anymore.

On the above image, I superimposed the scatter plot from OpenAI blogpost and my estimates of compute required for some recent large and ambitious ML experiments. To the best of my knowledge (and I have tried to check for this), there haven't been any experiments that required more compute than those shown on the plot.

The main thing shown here is that less than one doubling of computational resources for the largest training occured in the 3-year period between 2018 and 2021, compared to around 10 doublings in the 3-year period between 2015 and 2018. This seems to correspond to a severe slowdown of computational scaling.

To stay on the trend line, we currently would need an experiment requiring roughly around 100 times more compute than GPT-3. Considering that GPT-3 may have costed between $5M and $12M and accelerators haven't vastly improved since then, such an experiment would now likely cost $0.2B - $1.5B.
# The Meaning Crisis

This Sunday at 12pm PT, Vaniver will give a talk about the "Meaning Crisis" – what previous solutions have looked like, and what new solutions might look like. This is informed by the lecture series by John Vervaeke, previously [discussed here](https://www.lesswrong.com/posts/i4BYp9xZeEebhg7qM/lecture-club-awakening-from-the-meaning-crisis).

We'll be meeting in Gather Town:

[http://garden.lesswrong.com?code=CQ1l&event=the-meaning-crisis](http://garden.lesswrong.com?code=CQ1l&event=the-meaning-crisis)
# How do scaling laws work for fine-tuning?

The scaling laws, at least according to the interpretation [used in Ajeya's framework](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines?commentId=7d4q79ntst6ryaxWD) (and this seems to be basically endorsed by tons of people I respect on this matter) say basically that if you increase parameter count by an order of magnitude, you also need to increase training steps/data points by about an order of magnitude, or else you are wasting your compute and could get the same performance with a smaller parameter count. For example, for a 10^14 parameter model (the size of the human brain, basically) we'd need 10^13 training steps/data points.

Now we have papers like [this one](https://bair.berkeley.edu/blog/2021/03/23/universal-computation/) claiming that pre-trained transformers can be fine-tuned to do well at completely different tasks (incl. different modalities!) by only modifying 0.1% of the parameters.

Does this mean that this fine-tuning process can be thought of as training a NN that is 3 OOMs smaller, and thus needs 3 OOMs fewer training steps according to the scaling laws? I'm guessing the answer is no, but I don't know why, so I'm asking.

(If the answer is yes, how does that not contradict the scaling laws for transfer [described here](https://arxiv.org/abs/2102.01293) and used in this [calculation by Rohin](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines?commentId=LqZidQ6bEtDnqP6ji)?)
# About Me: Background and Biases

  
I've been a civil servant working in state government for almost ten years. I've worked at several regulatory agencies, I take lots of [civil service tests](https://www.cs.ny.gov/jobseeker/) to ([among other things](https://www.lesswrong.com/posts/nGExy2RtLi3YuitxC/about-me-background-and-biases?commentId=HMfRtTBY6T5ohwgG8)) manage my nagging [impostor syndrome](https://en.wikipedia.org/wiki/Impostor_syndrome), and as a result I've also had the opportunity to interview for jobs in this leviathan I expect to be connected to for the foreseeable future. Even when I didn't get the jobs I interviewed for, it's always given me an opportunity to "pull back the curtain" on a lot of the under-appreciated things that government does (sometimes unbeknownst to average citizens).   
  
Currently, I work as a paralegal, but I dabble with actuarial science (I passed [Exam P](https://www.soa.org/education/exam-req/registration/edu-registration/) and I'm currently studying for Exam FM). Before I started in state government I temped at an insurance company. In what now seems like prehistoric times I pursued a PhD in math, managed to pass my qualifying written and oral exams, completed a master's degree, but never completed a dissertation. Cognoscenti refer to this as "ABD" ("all but dissertation"). I'm fortunate enough to be represented by a union that negotiated very good [educational benefits,](https://nyscseapartnership.org/) and that allows me to take courses that nourish my mind.   
  
This gives me some insight that others might not have, but inevitably it may also bias me somewhat. Specifically, I may have higher standards than most for what "good" government systems ought to look like, particularly those that leverage information technology. Thus, while I think we might all broadly agree on [Gehm's contrapositive of Clarke's Third Law,](https://web.archive.org/web/20041229021232/http://fanac.org/fanzines/MT_Void/MT_Void-2319.html) to wit:

> Any technology that does not appear magical is insufficiently advanced.

I am *also* quite willing to concede that some things that might appear magical to you, dear reader, may not seem quite as magical to me - and vice versa. In some cases the best thing we can do is let everyone else catch up.   
  
Hope this helps.
# What do you think would be the best investment policy for a cryonics trust?

Assume this policy will be followed by someone else, potentially for centuries, unless your cryonics revival timeline is shorter.

You can answer differently for different trusts; notably:
1) Trust with the goal of maintaining a patient in stasis indefinitely, and providing the fund for revival
2) Trust with money that you will gain access to again once you're reanimated
# Risk Budgets vs. Basic Decision Theory

I've been wondering about the decision theory of having a "risk profile", e.g. "200 microcovids per week", and at least under some simplistic but not unreasonable assumptions I don't think it makes sense.  
  
Let's say we just have one activity A which yields some fixed positive utility *U* but has a certain probability *p* of giving you covid which has a large cost *C*. Then the expected utility of A is simply *E = U - pC*. If *E > 0* then you should just do A as much as possible, otherwise not at all. There's no cap on total covid risk. (OTOH the "risk profile" method would recommend doing A about 200e-6/*p* times.)  
  
One counterargument is that the positive utilities don't add up linearly. That is, just because going to a restaurant once is nice, doesn't mean you'd enjoy constantly being in one. But I think this is a red herring -- you could instead imagine a variety of different activities (movies, dating, eating out, parties, etc.) that do add up more or less linearly. You could also just be the type of person who gets the same enjoyment from eating out every day. Either way, the issue of "where does microcovid budget come from?" remains.  
  
I think the crux of the matter is that (consequentialist) decision theory is Markovian. When you make a decision, you only care about the state you're in right now, not in how you got there. So whether or not you did some risky activity yesterday, unless it actually gave you covid, there's no effect on your current state, and so therefore it shouldn't affect your current and future decisions about whether to engage in something risky.  
  
To be clear, my goal isn't to abandon risk budgets -- they seem very sensible to me, and I don't have a good replacement. But, I'd like to know if there is some better model which captures the intuition around risk budgets (or an error in the above reasoning).
# Preventing overcharging by prosecutors

In criminal cases in the United States prosecutors often add a lot of charges for a defendant to have ammunition for coercing the defendant into a plea deal. This is toxic because the defendant doesn't know which of those charges are likely to hold up in court if the case would be decided by a jury. Given that there's little cost to the prosecutor for adding additional charges, defendants are often overcharged.

I propose that whenever a prosecutor files a charge for a defendant, the prosecutor should state the likelihood that in the absence of a deal the court will find the defendant guilty of the charge. The ability of the prosecutor to accurately access the likelihood can be measured via the Briers score or a Log score.

The current score should be publicly accessible on the website of the court. This allows the defendant to know whether they can trust the likelihood values the prosecutor gives. The score should also be printed on ballots when the prosecutor seeks reelection to create much higher incentives for the prosecutor to give the correct likelihood then convicting a lot of people.

After the prosecutor provides the likelihood for the charges it's much easier for a defendant to make a good decision about whether taken a given plea deal is in their interest. Prosecutors with a good Briers score will be able to make more plea deals to reduce their overall workload because it's easier for the defendant to know that a deal is in their interest.

While this reform wouldn't fix all problems with plea deals, as some plea deals are due to the defendant being given charges that would actually hold up in court given the existing criminal code, the reform will provide defendants with fairer plea deals. Defendants getting fair plea deals is good for the system given that it keeps overall legal costs down.

I would expect that many juries will automatically throw out a 10% or 20% charges because there will be people on the jury who would argue that the prosecutor thinking that there's only a 20% chance that a charge holds means that there's reasonable doubt against the charge.

This reform is both in the interests of citizens who care about law and order and citizens who care about reducing sentencing overall as everybody should be interested in prosecutors providing fair plea deals.
# The Accord

In the middle of last year, I wrote a transhumanist sci fi novella inspired by (among other things) SSC's post about the Archipelago. It was quite intentionally written as a sci fi story drawing on Less Wrong discussions of AI, and it uses an idea I got from something EY wrote about AI merging their value functions to create the government of the system. It also draws ideas from Isaac Arthur's futurism youtube channel (which I also think a lot of people here might like). 

Also, it isn't really a traditional novel or story, rather it is basically one of those books written by a fake historian that the quotes at the start of each chapter in a sci fi or fantasy novel come from. That and excerpts from the comment section after each chapter. I try looking at the archipelago as an example of one way to distribute resources in a post human labor society. But it definitely not a post scarcity world, since you can't get that if while you have physical limits to the total amount of energy and stuff you can use. So I'm also exploring the ways that malthusian limits might manifest in the context of an AGI that grabs all of the resources available and makse them available for humans to use, but which also fully respects human autonomy 

So I've posted the whole thing for free, so go read it if any of that sounds cool, :)

[The Accord](https://timunderwoodscifi.wordpress.com/2020/07/18/introduction-and-crush-habs/)
# If my previous research is wrong, what are my options ?

In 2007 I coauthored a scientific paper, which is essentially a method for food analysis. To be honest, we have noticed one sample behaving weirdly already at that time, but we rationalised it away and omitted the weird sample from the results. Still, just in case, we have burried the article in a local journal of our institution, which is low profile. Later I discovered, it is wrong by design (My design :-( ). Knowing more background would have prevented the problem, but I did it as a side job. I discovered the full stupidity of the design in cca 2012 and told my boss. He suggested we correct ourselves by publishing another paper, claiming we did more experiments and found out we were wrong. I agreed, obtained more samples, my colleague worked on it, but I went on a long maternal leave with 2 kids and forgot to follow up on it. It is uncorrected up until this day. What are my options ?

If we publish a new article, as my boss wanted, I fear some people will still find the first paper and not the second one, will keep quoting it, and, god forbid, use that published method. (To my best knowledge the method was not used so far. I hope it really wasn't). Retraction would be the cleanest way to do it, but also the worst looking. Also we could publish erratum. Erratums are paired up with the original articles and search engines show them together. But can erratum contain new samples ? Any other ideas ?

If my boss finds the correction a low priority, as he did so far, can I somehow act alone ?

Can I have any problems, that I did not correct myself fast enough ? I left some traces about knowing the design is wrong in 2012. I wrote the truth to one lady requesting the reprint. I also wrote a book chapter, explaining how important it is to know the backround while designing analysis method, and giving the example of this tricky food. I did not mentioned nor quoted my article...

Please, don't be too judgemental, and tell me, how you would or did handle these situations.
# Open and Welcome Thread - April 2021

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1). If you want to orient to the content on the site, you can also check out the new [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Is there any plausible mechanisms for why taking an mRNA vaccine might be undesirable for a young healthy adult?

We haven't seen any problem in the short term, but is the probability of long term negative side effects (given it's a new technology) sufficiently unlikely for a young healthy person to take it?

Am particularly concerned with unknowns long-term future negative side effects, but which we might be able to estimate from first principles, or through some more abstract reference class, or something.

I already fully isolate, and would only value my freedom a bit, whereas I value my health a lot.
# Aging: A Surprisingly Tractable Problem

Aging kills more than twice as many people as all the other things that kill people combined, yet anti-aging research remains shockingly neglected. The scientific feasibility of anti-aging has been confirmed by many world-leading experts such as Professor David Sinclair from Harvard Medical School. However, while billions of dollars of governmental funding is directed towards treating the diseases of aging, only a tiny fraction is spent on attempting to create treatments for the root causes of these diseases. In this discussion, we will introduce the scientific feasibility of anti-aging technologies, and the ethical impacts of these developments. Will aging lead to immortal dictators? Distributional justice issues? Overpopulation? These questions will be addressed in this eye-opening presentation.

Jack Harley is a biomedical scientist and biotech analyst with a focus on anti-aging biotechnologies. After graduating from neuroscience and bioethics at the University of Oxford, Jack has served as Vice President of the Oxford Society of Ageing and Longevity, and worked in several longevity biotechnology companies based in Australia and the US. Jack is a long time Effective Altruist, and his involvement began in 2014 after meeting Peter Singer. Recently, Jack has outlined the rationale for anti-aging in a popular article published on the rationality forum LessWrong in 2020 (https://bit.ly/3v5wL2H). Jack’s vision is to encourage greater investment and scientific talent into longevity biotechnology to extend the healthy lifespan of billions of people alive today.
# Identifiability Problem for Superrational Decision Theories

*Superrationality, and generalizations of it, must treat options differently depending on how they're named.*

Consider the penny correlation game: Both players decide independently on either head or tails. Then if they decided on the same thing, they each get one util, otherwise they get nothing. You play this game with an exact copy of yourself. You reason: since the other guy is an exact copy of me, whatever I do he will do the same thing. So we will get the util. Then you pick heads because its first alphabetically or some other silly consideration, and then you win. How good that you got to play with a copy, otherwise you would have only gotten half an util.

Now consider the penny anti-correlation game: Both players decide independently on either head or tails. If they decided the same thing, they get nothing, otherwise they get one util each. You play this game with an exact copy of yourself. You reason: since the other guy is an exact copy of me, whatever I do he will do the same thing. So the best I can do is to pick randomly with 50% chance, that way I get half an util. Thats if the gamemaster is nice. If he isnt nice, then identical copies in identical environments get the same result from their RNG. In that case you lose the game whatever you do. How bad that you had to play with a copy, otherwise you could have gotten half an util.

The problem is that these are the same game, only with the labels for one players actions switched (or for the other player. which is exactly the problem). Despite this, superrational reasoning gives us different results. This could happen because we have taken a theoretical symmetry from a physical one, and physical symmetry can pay attention to the detailed makeup of symbols. If we are aware of the difference, it should not be too surprising that there is a setup exploiting it. 

Now I think the reasoning presented is correct in both cases, and the lesson here is for our expectations of rationality. From what I've seen people formed their expectations about what algorithm-controlling decision theories will do when they are worked out largely around rationality conditions, and what *ought* to be the outcome. In this way, common knowledge of rationality was supposed to guarantee various things, like pareto-optimal equilibria or an existent and unique solution concept. This has taken a blow here. Superrationality, from which we originally tried to generalize these properties, doesn't provide them even for the symmetrical games where it is applicable. Furthermore, this is a strong example of a classical rationality condition being violated, and in a way that is unfixable in principle. *No matter* what rationality ends up being, you will not win the anti-correlation game with your copy, and you will win the correlation game, even though they only differ by a renaming.
# Ben Goertzel's "Kinds of Minds"

I'd like to recommend Ben Goertzel's book from 2006  [*Hidden Pattern*](https://drive.google.com/file/d/1zaq_-FCWpushbtxLvemSU97HS0klVyb0/view?usp=sharing), and particularly Chapter 2 ["Kinds of Minds"](https://drive.google.com/file/d/1ISibMWTBuMZ4Qu1vG4BLlRuZCrEJkOso/view?usp=sharing). (The book is also [available at](https://goertzel.org/HiddenPattern_march_4_06.pdf) the author's site, but that may trigger SSL certificate warnings.)

We've come to understand that a future AGI may have non-anthropomorphic  implementation and behavior, but this chapter situates possible minds in a variety of types of implementation and embodiment that I have not seen described elsewhere.

Specifically, p.17 compares

> *   High-resource minds (like AIXI) or modest-resource

and among modest-resource minds

> *   Embodied versus non, and within the "embodied" category: singly versus multiply versus flexibly embodied, and tool-dependent versus non
> *   Mindplexes versus fully unified minds
> *   Socially-dependent versus non

If there is other literature that enumerates and compares these and other kinds of minds, I'll be glad to know.
# What if AGI is near?

Consider the following observations:

*   The [scaling hypothesis](https://www.gwern.net/newsletter/2020/05#scaling) is probably true: we will likely continue to see great improvements in AI capabilities as model sizes increase.
    *   Sutskever [mentioned](https://www.reddit.com/r/ControlProblem/comments/kpcmxn/the_best_models_of_2021_will_make_the_best_models/) that models currently under development already have dramatic and qualitative improvements (e.g. going more multimodal) over those already made in 2020.
*   AI model sizes are likely to increase very quickly over the short term.
    *   [NVIDIA’s CEO](https://old.reddit.com/r/ControlProblem/comments/mpsdbc/we_expect_to_see_models_with_greater_than_100/): “We expect to see models with greater than 100 trillion parameters by 2023". Something 1000x the size of GPT-3, given its already shocking performance, is scary to imagine, to say the least.
    *   Even if OpenAI is cautious and will not go ahead with potentially catastrophic projects (dubious), the wide availability and rapidly decreasing cost of efficient hardware, along with publicly available information about how to train GPT-like architectures, means that some organization will achieve scaling.
*   We will likely not be able to solve AI alignment within the next few years, even if AI safety research were to speed up dramatically.
    *   Deep learning AIs in the near term cannot conceivably be anything besides not remotely aligned.

Questions:

*   What takeoff speeds are likely with large deep learning models, if they attain AGI? Are medium takeoffs more plausible due to deep learning type AIs having less obvious “tweaks” or ways to recursively self improve by altering code?
    *   The key extension of that question is how will society react to presence of AGI? will it even shut down, stop development & wait for safety?
    *   Perhaps takeoff speeds are moot beyond human level, because even a human-level AGI would have the wits to proliferate itself over the internet to computers all over the world and therefore make its eradication impossible once it has come into existence? Therefore once it has been created, it would be a slow but certain, inexorable death?
*   Given short term AGI, what options are available to reduce the risk of existential catastrophe? Would global nuclear war or a similarly devastating event that prevents technological progress be the only thing which could stave off AGI annihilation in the near term?

More serious thought needs to be given to this, to solemnly consider it as a looming possibility.