Harley-ml/lesswrong huggingface dataset:
# What happens to variance as neural network training is scaled? What does it imply about "lottery tickets"?

Daniel Kokotajlo [asks](https://www.lesswrong.com/posts/wFJqi75y9eW8mf8TR/does-the-lottery-ticket-hypothesis-suggest-the-scaling) whether the lottery ticket hypothesis implies the scaling hypothesis.

The way I see it, this depends on the distribution of "lottery tickets" being drawn from.

*   If the quality of lottery tickets follows a normal distribution, then after your neural network is large enough to sample decent tickets, it will get better rather slowly as you scale it -- you have to sample a whole lot of tickets to get a really good one.
*   If the quality of tickets has a long upward tail, then you'll see better scaling.

However, a long tail also suggests to me that variance in results would continue to be relatively high as a network is scaled: bigger networks are hitting bigger jackpots, but since even bigger jackpots are within reach, the payoff of scaling remains chaotic.

(This could all benefit from a more mathematical treatment.)

So: what do we know about NN training? Does it suggest we are living in [extremistan or mediocristan](https://en.wikipedia.org/wiki/The_Black_Swan:_The_Impact_of_the_Highly_Improbable#Part_one:_Umberto_Eco's_anti-library,_or_how_we_seek_validation)?

Note: a major conceptual difficulty to answering this question is representing NN quality in the right units. For example, an accuracy metric -- which necessarily falls between 0% and 100% -- *must* yield "diminishing returns", and *cannot* be host to a "long-tailed distribution". Take that same metric and send it through an inverse sigmoid, and now you might *not* have diminishing returns, and *could* have a long-tail distribution. But we can transform data all day. The analysis shouldn't be too ad-hoc. So it's not immediately clear how to measure this.
# Tags Discussion/Talk Thread

The LessWrong dev team is hard at work creating Talk Pages/Discussion pages for tags. When they're done, every tag page will have a corresponding talk page which lets users discuss changes and improvements related to that tag.

We don't have that yet, so in the meantime, please make comments you have about tags (generally or for specific-tags) here. If you're talking about a specific tag, of course, make sure to link to it. You might also want to link back to your comment in the body of the tag description, e.g., "Tag Discussion here"

Examples of things you might comment about a tag:

*   Wow, this is a great tag! / I think this tag makes no sense.
*   I propose renaming this tag to X for clarity.
*   I want feedback on these proposed changes to this tag description.
*   I am confused why <post> has been tagged under this tag.
*   I have made several improvements. Mods, please review this [tag's grading](https://www.lessestwrong.com/tag/tag-grading-scheme). I think it is now a B-Class tag.

Or:

*   Why doesn't a tag for Z exist?
*   I really want <feature>, that would make my life much better.
*   <Thing> seems broken.

Also, feel free to use this space to claim credit for tags you've worked hard to make great! (we'll give you karma!)

Other relevant pages about tagging
----------------------------------

*   [Tagging FAQ](https://www.lesswrong.com/posts/E6CF8JCQAWqqhg7ZA/tagging-faq)
*   [Tag Grading Scheme](https://www.lesswrong.com/tag/tag-grading-scheme)
*   [Tags Spreadsheet](https://docs.google.com/spreadsheets/d/1Oiv_Mg_7mEhP0Ik6Bs1V99G2v4eevL-LvkxQfD-blqw/edit#gid=791972812) (GSheet, good way to find tags to work on)
# Predictions for GPT-N

Regarding GPT-3, there is [some discussion whether growing the model would transform it into an Oracle AI](https://www.lesswrong.com/posts/3nDR23ksSQJ98WNDm/developmental-stages-of-gpts). I looked into the actual benchmark results (Appendix H in [the paper](https://arxiv.org/abs/2005.14165v4)) to see if we can predict something useful from the actual measurements.

**Method:** The OpenAI team ran a suite of 63 different benchmarks (including sub-types), each for zero/one/few shot. In each scenario, there are 8 model sizes. I looked at how results scale with model size. With only 8 measurements, there is a large associated uncertainty for predictions. Formally, one would test the trend function using a  
Bayesian model selection between a linear and (e.g.,) a polynomial. I did this for a few and then eye-balled the rest. So, please take the following as an indication only.

**Disclaimer:** The smallest model for GPT-3 has  108 parameters, the largest 1011. That's a span of 3 orders of magnitude. Scaling this out to many more orders of magnitude is dangerous. Thus, take these numbers only as an indication.  
  
**Results.** For the following tests, I find an **asymptotic trend**. Scaling the model will apparently not yield fantastic results for:

*   HellaSwag, LAMBADA, PIQA, CoQA, OpenBookQA, Quac, RACE, CB, ReCoRD, WiC
*   Translations - but unclear level description.

In the following tests, it is **unclear if the trend is asymptotic** or better than that:

*   SAT: Could be linear, could be asymptotic. If linear, it will achieve 100% at 1016 parameters.
*   StoryCloze, Winograd, Winogrande, SQuADv2, DROP, Copa.

**These tests show a linear scaling:**

*   TriviaQA (1013 parameter estimate to achieve 100%)
*   BoolQ (1015)
*   MultiRC (1016)
*   ARC (1016)
*   SuperGLUE (1018)
*   WSC (1020)
*   WebQs (1021)
*   Cycled (1023)

**Some tests scale neither linear nor asymptotic:**

*   Symbol: Near exponential (1012)
*   Arithmetic: Exponential; one-digit composite may achieve 100% at 1014
*   Reversed: Near exponential (1016)
*   Anagrams: Polynomial (1019)
*   ANLI: stepped, unclear
*   RTE: stepped, unclear

  
**Summary:** About half of the tested skills will likely not scale much with larger models. The other half will (e.g., TriviaQA, SuperGLUE, arithmetic, anagrams). Going to e.g., 1016 parameters - would that make an Oracle AI? Probably it's not sufficient, but I'm interested in hearing your opinion!
# Is the work on AI alignment relevant to GPT?

I see a lot of posts go by here on AI alignment, agent foundations, and so on, and I've seen various papers from MIRI or on arXiv. I don't follow the subject in any depth, but I am noticing a striking disconnect between the concepts appearing in those discussions and recent advances in AI, especially GPT-3.

People talk a lot about an AI's goals, its utility function, its capability to be deceptive, its ability to simulate you so it can get out of a box, ways of motivating it to be benign, Tool AI, Oracle AI, and so on. Some of that is just speculative talk, but there does appear to be real mathematics going on, for example on embedded agency. But when I look at GPT-3, even though this is already an AI that Eliezer finds alarming, I see none of these things. GPT-3 is a huge model, trained on huge data, for predicting text. That is not to say that it cannot be understood in cognitive terms, but I see no reason to expect it to be. It is at least something that would have to be demonstrated before any of the formalised work on AI safety would be relevant.

People speculate that bigger and better versions of GPT-like systems may give us some level of real AGI. Can systems of this sort be interpreted as having goals, intentions, or any of the other cognitive and logical concepts that the AI discussions are predicated on?
# PSA: Tagging is Awesome

I'd like to supplement the [open call for taggers](https://www.lesswrong.com/posts/uqXQAWxLFW8WgShtk/open-call-for-taggers) with a few points:

*   **Tagging is a neglected cause area.** There is still a huge amount to do, and tagging makes a real difference in making LessWrong content easier to discover, explore, and re-find.
*   **The problem is real.** People find LessWrong difficult to read because it is full of deep inferential distances and special jargon. Tags offer an easy way to disambiguate jargon, and reference all the relevant material on a subject.
*   **Tagging isn't just altruistic.** Want to promote an idea/topic? Tagging posts and writing a good tag description is a great way to make that topic easier to discover and explore. If writing blobs of text which lots of people later read pumps your ego, tagging is a good way to do that. Write that tag before someone else! But it's also useful -- not just to other people, but also, to yourself. Tagging posts on subjects you love, and upvoting the tag on the most relevant ones, will make it easier for you to reference them later.
*   **You will probably discover things you want to read.** Tagging naturally gets you searching for content on LessWrong related to your favorite topics. You are likely to discover than more has been written on these topics than you previously realized.
*   **Tagging is easy.** Whenever you think of a tag you want to exist (usually because you're reading a post and decide to tag it with something, only to discover the tag doesn't exist yet), just do a search for that thing on LessWrong and tag all the relevant results! There are other approaches, of course, but if everyone did this, then we'd be in a pretty good position: any new tag created would already be put on most of the relevant posts. (This strategy doesn't work for tags which cover a significant portion of the content on LessWrong, such as [the core tags](https://www.lesswrong.com/posts/KnHPrP2K8SewomJY6/the-core-tags), of course.)
    *   If you're not sure what to tag, take a look at the [top posts without tags](https://www.lesswrong.com/allPosts?filter=untagged&sortedBy=top&timeframe=allTime). You may want to familiarize yourself with [the core tags](https://www.lesswrong.com/posts/KnHPrP2K8SewomJY6/the-core-tags) and the [concepts portal](https://www.lesswrong.com/tags/all), so that you're not missing some obvious ones when you tag things.
# Sunday August 2, 12pm (PDT) — talks by jimrandomh, johnswenthworth, Daniel Filan, Jacobian

This Sunday at 12pm (PDT), we're running another session of "lightning talks" by curated LessWrong authors (see [here](https://www.lesswrong.com/tag/lesswrong-events) for previous weeks' transcripts).

*   Each talk will be 3-5 minutes followed by discussion. Afterwards, we'll have a hangout in breakout rooms. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
*   We want to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

*If you're a curated author and interested in giving a 5-min talk at a future event, which will then be transcribed and edited, sign up* [*here*](https://forms.gle/iwFatbhys9muPmQA7)*.*

Speakers
--------

*   **jimrandomh**
    *   Curated posts: [A Significant Portion of COVID-19 Transmission Is Presymptomatic](https://www.lesswrong.com/posts/9GyKccaJdLEbdhyTi/a-significant-portion-of-covid-19-transmission-is)
*   **johnswentworth **
    *   Curated posts: [Gears-Level Models are Capital Investments](https://www.lesswrong.com/posts/nEBbw2Bc2CnN2RMxy/gears-level-models-are-capital-investments), [Everyday Lessons from High-Dimensional Optimization](https://www.lesswrong.com/posts/pT48swb8LoPowiAzR/everyday-lessons-from-high-dimensional-optimization), [Interfaces as a Scarce Resource](https://www.lesswrong.com/posts/hyShz2ABiKX56j5tJ/interfaces-as-a-scarce-resource), ...
*   **Daniel Filan**
    *   Curated post: [Insights from 'The Strategy of Conflict'](https://www.lesswrong.com/posts/2SeN2MjmMzZB25hBo/insights-from-the-strategy-of-conflict)
*   **Jacobian**
    *   Curated posts: [Winning is for Losers](https://www.lesswrong.com/posts/DhQkDgLiYe28P3LHM/winning-is-for-losers), [Seeing the Smoke](https://www.lesswrong.com/posts/5okDRahtDewnWfFmz/seeing-the-smoke)

Details
-------

**When?** Sunday August 2, 12pm (PDT)

**Where?** [Zoom](https://us02web.zoom.us/j/87099956068).
# Can you gain weirdness points?

A useful idea I've been looking into more lately, is "weirdness points". Basically, some good ideas are really unusual. People are often [biased against unusual ideas](https://www.lesswrong.com/posts/atcJqdhCxTZiJSxo2/talking-snakes-a-cautionary-tale). It's often seen to be easier to fight for one weird thing, than to fight for multiple weird things. Therefore, we ought to prioritize what we fight for, so the most important things get our weirdness points, and [tamper down on the weirdness](https://www.lesswrong.com/posts/wkuDgmpxwbu2M2k3w/you-have-a-set-amount-of-weirdness-points-spend-them-wisely) in other areas of our lives/work.

Typical social explanations of weirdness points aren't completely helpful. Power, status, and wealth would seem to bestow weirdness points. But politicians, celebrities, and wealthy people aren't always as free from the weirdness constraint as would be guessed.

Maybe communities and media are fracturing so much that weirdness points are more dependent on your community than your actions. (The social psych idea, "[idiosyncrasy credits](https://en.wikipedia.org/wiki/Idiosyncrasy_credit)", is defined in terms of a _group's_ expectations, not those of society-at-large or people-who-are-reachable-but-not-already-on-your-side.)

Weirdness points seem like a valuable (and limited) resource, especially if you are promoting or enacting multiple ideas (A.I. safety _and_ improving rationality _and_ open borders, for example). As with anything valuable to our goals, we ought to figure out if we can get it, and at what cost.

So, the questions for discussion:

*   What actually determines weirdness points?
*   Are weirdness points important or useful or accurate, as a predictive model? How constrained are people's actions, really, in promoting weird ideas? In what contexts?
*   How can one gain weirdness points?
*   Has anyone at any time "hacked" weirdness points, by successfully promoting multiple weird things / having weird habits / having a weird personality, _without_ eventually running their status/credibility into the ground? (The only person I can think of offhand _might_ be [Richard Feynman](https://en.wikipedia.org/wiki/Richard_Feynman).)
# Food Spending During Covid

Our house has [shared groceries](https://www.jefftk.com/p/shared-house-setup), and in my most recent reconcile I saw that our spending is up quite a bit relative to this time last year. In 2019, April through July, our monthly spending per person was $172, while in 2020 it was $303. In the big picture this is due to the pandemic, but I'm not sure which factors are responsible for most of the change. Some ideas:

*   Our house is [not going into stores](https://www.jefftk.com/p/partially-stepping-down-isolation), and we've been having groceries delivered. This reduces our risk, but it's also somewhat better at limiting spread in general. Grocery delivery is more expensive, especially when you include the tips. I expect this is the largest component for us.
    
*   \[EDIT: Julia points out that another reason Instacart is more expensive is that we're generally choosing the cheapest option for things, and so when they make substitutions they're picking more expensive versions or generally not predicting our preferences very well.\]
    
*   Food has gotten more expensive, but only by [~3%](https://fred.stlouisfed.org/series/CPIFABSL): CPIFABSL is up to 269 from 261 in February.
    
*   I'm not eating lunch at work, since the office is closed. I also used to eat my fruits and vegetables there, and they're more expensive than the mostly carbs I used to eat at home.
    
*   People in the house aren't eating out anymore. I don't think any of us were eating that many other commercially prepared meals, but it's a component. The house grocery budget won't count a restaurant meal, but it will count the food you eat at home because you didn't go out to eat.
    
*   I used to also buy things when they were on sale, and without walking through the grocery store that doesn't really work anymore.
    
*   We've been buying some things from a restaurant supply place, and while for some products it is substantially cheaper, others are fancier than what we would normally buy. I suspect this one about cancels out.
    

It's very noisy, but here's a chart showing our historical food spending. It's a 90 day running average, so the recent rising slope is mostly the step-change of our newer more expensive situation being smeared over a longer period.

[![](https://www.jefftk.com/per-person-food-spending-covid.png)](https://www.jefftk.com/per-person-food-spending-covid-big.png)
# What are you looking for in a Less Wrong post?

My model of "What LW as a community considers a good post" is not accurate enough for my taste. Sometimes I don't understand the karma/number of comments (or lack thereof) of some posts, and it seems rude or arrogant to ask (whether it's someone else's post or one of mine). Among other things, this might help me decide whether to spend the hours necessary to write a post, or if it's not the kind of ideas people are interested in here.

So I'm asking you, yes you, what makes a LW post that you want to read. That you want to upvote or strong upvote. That you might comment. A good post, basically.
# Burley Bee kids bike trailer review

With the coronavirus we haven't been taking buses or taxis, which had mostly limited us to walking distance with our kids. \[1\] Expanding to bicycling distance would help a lot, so we decided to get a bicycle trailer.

I did a bunch of reading on various forums, and it looked like the [Burley Bee](https://www.burley.com/product/bee/) was the best fit for us. It holds two children, with a combined weight of 100 lb, while being relatively light and sturdy. It has a rain cover, in case we get caught out in bad weather, and it doesn't convert into a stroller. No suspension, aside from the tires and the give in the fabric, but the kids haven't complained. There's a good amount of storage space in the back of the trailer. The safety flag is included.

[![](https://www.jefftk.com/burley-bee-bike-trailer-with-kids.jpg)](https://www.jefftk.com/burley-bee-bike-trailer-with-kids-big.jpg)

[![](https://www.jefftk.com/burley-bee-bike-trailer-with-bike.jpg)](https://www.jefftk.com/burley-bee-bike-trailer-with-bike-big.jpg)

It comes with one bicycle hitch, but it's a very simple part so I bought [additional ones](https://smile.amazon.com/gp/product/B07FNKLK6N/) so that me, my wife, and our au pair could each have a hitch on our bicycles. The hitch goes onto the rear axle, on the left hand side, and is about 3/16" thick.

We've been using it for about 6 weeks now, and we haven't had any problems. The kids are happy with it, it's easy to bike with, and it generally seems well designed.

(When we decided we wanted to buy one, in late May I think, Amazon, the other big online stores, and our local bike stores were sold out. I did some looking, though, and found a non-local bike store that had some in stock and was happy to ship. At this point, however, it seems to be back in stock [at the manufacturer](https://www.burley.com/product/bee/) and [on Amazon](https://smile.amazon.com/Burley-Bee-Child-Trailer/dp/B089T61Z7D/).)

  
\[1\] We considered getting a car, but addition to being a lot more expensive, we have two housemates with cars that we can [occasionally borrow](https://www.jefftk.com/p/transportation-tracking). With covid their cars have mostly been sitting at home, so getting a third seemed excessive. But for something within a ~30min bike ride of the house a car seems like overkill, and it's not worth the coordination with the car owner / using a favor.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100177255615162)*
# Why isn't there a LessWrong App? Is a "blog-app" sustainable/useful for a community?

I am a newbie lesswronger, so it might be an already asked question, but I can't find an answer in the FAQs. I also don't have any idea of how many resources are needed to support a blog-app, given an already developed site/community. I would be glad if someone could explain the issues/difficulties, and why the community never exploited this channel to acquire new users/better accessibility (I would love to have an app instead of using the browser to browse the site).
# What are the most important papers/post/resources to read to understand more of GPT-3?

I'm way more used to thinking about weird maths or distributed algorithms or abstract philosophical problems than about concrete machine learning architectures. But based on everything I see about GPT-3, it seems a nice idea to learn more about it, even if only for participating in the discussion without spouting non-sense.

So I'm asking for what you think are the must-reads on GPT-3 specifically, and maybe any requirement to understand them.
# Solving Key Alignment Problems Group

I want to map out all the open problems in Alignment; understand the key topics, how they relate and how the parts form a greater whole. I believe a small group of people who can _notice_ _when they're confused_,  can make substantial conceptual progress on many of the open problems. Besides, it sounds like a lot of fun.

To do this, I will have a 1-hr open meeting every week working on this. If you're reading this, you are free to join.

But wait, what would this look like?
====================================

Once a week, time TBD, we would call and go through a few questions, 5-minute timer style. Then discuss. For example:

Week 1:

1\. Map out all of alignment from a first principles perspective. \[Your solution should feel like it reasonably solves alignment. It can contain many "black boxes" that do the hard work, so spend extra time on opening up the black boxes and exploring them.\] (30 minutes, then discuss)

2\. Design images that explain the same information from 1 above. \[You can use whatever tool you want, I will use draw.io\] (10 minutes, then discuss/ post images in Discord)

Week 2: Vote on a specific topic to zoom in on and explore from first principles

Week 3: Literature Review of topic from Week 2

Week 4 on: TBD

\[Specific questions and phrasings will change weekly based on feedback\]

What if I won't be helpful?
===========================

Discussion will be on a volunteer basis. I expect there will even be _too_ much talking (or not enough time to talk). You will not be expected or called out to tell everyone your answer. You can even write up your thoughts in the Discord throughout the week if that's how you'd like to contribute.

One hour/week won't amount to much
==================================

As mentioned, there is a Discord server attached. There will always be open problems to explore and work on the week between meetings. If you want to join, you will NOT be expected to work on anything in-between meetings, but I will (I only recommend you do that if you find the topic fun or interesting).

I would love to have a community of people focused on a specific topic, discussing and giving feedback. To me, a Discord server is useful for creative work since I have a lower standard for anything I write (as opposed to writing this post).

What if you waste time on dead ends?
====================================

I will post weekly on what we figured out in the last meeting, as well as post the questions for the next week. I am counting on the LW community to provide feedback, to hopefully point out faulty assumptions and dead-end research directions.

Logistics
=========

If you're interested, pm me here on LW or email me at logansmith5(at)gmail. You will get a WhenToMeet to pick a time and a Discord server link. We will start the week of Aug 9th-15th, depending on the decided time.
# A sketch of 'Simulacra Levels and their Interactions' 

Two sketches i made based on [Simulacra Levels and their Interactions](https://www.lesswrong.com/posts/qDmnyEMtJkE9Wrpau/simulacra-levels-and-their-interactions). These are just sketches right now, i intend to make something better looking in the future (this is especially true for the first image). but i'd love to hear ideas and get feedback on these early versions.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7105e0dd877a41eead0599b792c0cc3242879384c5768db8.png)

The above diagram would look much better if there was symmetry. but the post misses some combos, some of which i also can't see how they're applicable (L1 & L2, for example).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e36a2f72ea5bed7efc6c6408d54d2fddbf36c43fe526c7b7.png)

The colors seem too happy for the topic to be honest :)

Here i also made a Venn version. it has [my idea for the ideologue](https://www.lesswrong.com/posts/qDmnyEMtJkE9Wrpau/simulacra-levels-and-their-interactions?commentId=H7oJarxjRoFGvTeyj), and then the only ones missing are (1, 3 & 4 VS 2) and (1 & 4). for the former i have a hard time thinking of something that would fit there, and for the latter I'm pretty sure there's isn't something that fits there.
# RT-LAMP is the right way to scale diagnostic testing for the coronavirus

**RT-LAMP is the right way to scale diagnostic testing for the coronavirus**

_by Kevin Fischer_

_kfischer $& gmail *( com_

[RT-LAMP](https://en.wikipedia.org/wiki/Reverse_Transcription_Loop-mediated_Isothermal_Amplification) compared to [RT-PCR](https://en.wikipedia.org/wiki/Reverse_transcription_polymerase_chain_reaction) is a less versatile and more recently invented laboratory technique. LAMP is more suited to actual diagnostic testing at scale than PCR but never became industry standard despite its demonstrated superiority. Labcorp/Quest/One Medical use PCR for their coronavirus diagnostic tests and they currently have delays as long as two weeks for issuing results which is totally unacceptable for fighting this pandemic. Color in San Francisco has one of few widely available [RT-LAMP tests for the coronavirus](https://www.color.com/wp-content/uploads/2020/05/LAMP-Diagnostic-Assay.pdf) and they generally give results back in one to two business days and are not bottlenecked by laboratory procedure.

An open access RT-LAMP coronavirus diagnostic toolkit was published at the end of July and I believe presents a solution to scaling diagnostic testing to where it needs to be. **_[A rapid, highly sensitive and open-access SARS-CoV-2 detection assay for laboratory and home testing](https://www.biorxiv.org/content/10.1101/2020.06.23.166397v2)_**

What steps can we take to help national testing capacity switch to RT-LAMP instead of RT-PCR? There are parallel approaches here -- bottom up development suggests distributed citizen science RT-LAMP labs to fill in gaps in testing capacity, top down distribution suggests convincing the major industry players to devote resources to switching to RT-LAMP over RT-PCR. I'm going to take the bottom up approach; can anyone else figure out how to help Quest and Labcorp switch over to the superior testing method?

Does anyone else want to set up their own RT-LAMP operation? I'm going to give it a try and will write a guide on what to buy once we're operational. One of my business partners was an innovator in LAMP primer design over ten years ago and we still own the lab equipment needed for the simple procedure. Presumably as non FDA approved citizen science it will have to be "for research only" and not for diagnostic testing purposes but it's possible the FDA is being cooperative and that actual FDA licenses could be issued on a timeframe that is reasonable.
# Infinite Data/Compute Arguments in Alignment

*This is a reference post. It explains a fairly standard class of arguments, and is intended to be the opposite of novel; I just want a standard explanation to link to when invoking these arguments.*

When planning or problem-solving, we focus on the hard subproblems. If I’m planning a road trip from New York City to Los Angeles, I’m mostly going to worry about which roads are fastest or prettiest, not about finding gas stations. Gas stations are abundant, so that subproblem is easy and I don’t worry about it until harder parts of the plan are worked out. On the other hand, if I were driving an electric car, then the locations of charging stations would be much more central to my trip-planning. In general, the hard subproblems have the most influence on the high-level shape of our solution, because solving them eats up the most degrees of freedom.

In the context of AI alignment, which subproblems are hard and which are easy?

Here’s one class of arguments: compute capacity and data capacity are both growing rapidly over time, so it makes sense to treat those as “cheap” - i.e. anything which can be solved by throwing more compute/data at it is easy. The hard subproblems, then, are those which are still hard even with arbitrarily large amounts of compute and data.

In particular, with arbitrary compute and data, we basically know how to get best-possible predictive power on a given data set: Bayesian updates on low-level physics models or, more generally, approximations of Solomonoff induction. So we’ll also assume predictive power is “cheap” - i.e. anything which can be solved by more predictive power is easy.

This is also reasonable in machine learning practice - once a problem is reduced to predictive power on some dataset, we can throw algorithms at it until it’s solved. The hard part - as many data scientists will attest - is reducing our real objective to a prediction problem and collecting the necessary data. It’s rare to find a client with a problem where all we need is predictive power and the necessary data is just sitting there.

(We could also view this as an [interface argument](https://www.lesswrong.com/posts/hyShz2ABiKX56j5tJ/interfaces-as-a-scarce-resource): “predictive problems” are a standard interface, with libraries, tools, algorithms, theory and specialists all set up to handle them. As in many other areas, setting up our actual problem to fit that interface *while still consistently doing what we want* is the hard/expensive part.)

The upshot of all this: in order to identify alignment subproblems which are likely to be hard, it’s useful to ask what would go wrong if the world-modelling parts of our system just do Bayesian updates on low-level physics models or use approximations of Solomonoff induction. We don’t ask this because we actually expect to use such algorithms, but rather because we expect that the failure modes which still appear under such assumptions are the hard failure modes.
# Titan (the Wealthfront of active stock picking) - What's the catch?

[Titan](https://www.titanvest.com/) is a Y Combinator startup that launched in 2018 and aims to do for active investing what Wealthfront, Betterment and Vanguard have done for passive investing.

They pick a basket of 20 companies with $10B+ market cap which they believe are above-average long-term-focused investments relative to the whole S&P 500.

Originally, their stock picking was done via a deterministic process of copying what a group of top hedge funds were reporting that they were doing. I'm not sure if that's still the case.

Their [2018-2020 performance](https://cdn.titanvest.com/library/Flagship_Tearsheet_073120.pdf) has been 16.8%/yr (net of fees) compared to 10.0% for the S&P 500, and a higher Sharpe ratio (.77 vs .51).

My question is, what's the catch?

Here's my guess: They're buying high-quality companies at high prices. That's how they can expect to have steady market-beating returns for a few years, until momentum reverses and/or once-in-a-few-years risks play out, at which point their P/E multiples will shrink and they'll plunge all the way down to cumulative market-matching returns, and worse after subtracting their fees.

Their ["process"](https://www.titanvest.com/process/) page claims they look for a Warren Buffet style "Margin of safety":

> Valuation is important. We seek companies that are trading at a meaningful discount to our estimate of their long-term intrinsic value, with little to no risk of permanent capital impairment.

But I'm not convinced there's much substance to their use of this term.

More (vague) info about how they pick stocks [here](https://cdn.titanvest.com/library/Titan_Investor_Handbook.pdf).
# Open & Welcome Thread - August 2020

If it’s worth saying, but not worth its own post, here's a place to put it. (You can also make a [shortform post](http://www.lesslong.com/))

And, if you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1). If you want to orient to the content on the site, you can also check out the new [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new).
# How do you organise your reading?

Maybe this is just a personal problem, but I frequently find myself with just an absolute mess of things to read. Blog posts over several different sites, PDFs, paper books, podcasts and other content I intend to consume. I suspect a significant proportion of it falls through the cracks, not least when I have three separate browser windows open just so I can see the tabs. The LW standard of each post referencing at least 10 other posts in-text doesn't help, because each of them seem useful, but I don't want to disrupt the flow of the current article, so I open it in another tab, and so on and so on.

So, my question is: Do you have any useful tools, tips, or techniques for organising things you intend to consume?
# Sunday August 9, 1pm (PDT) — talks by elityre, jacobjacob, Ruby

This Sunday at 1pm (PDT), we're running another session of "lightning talks" by curated LessWrong authors (see [here](https://www.lesswrong.com/tag/lesswrong-events) for previous weeks' transcripts).

*   Each talk will be 3-5 minutes followed by discussion. Afterwards, we'll have a hangout in breakout rooms. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
*   We want to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

*If you're a curated author and interested in giving a 5-min talk at a future event, which will then be transcribed and edited, sign up* [*here*](https://forms.gle/iwFatbhys9muPmQA7)*.*

Speakers
--------

*   **elityre**
    *   Curated posts: [Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on?](https://www.lesswrong.com/posts/b9ErEMen42X94749t/does-it-become-easier-or-harder-for-the-world-to-coordinate)
    *   [How do you assess the quality / reliability of a scientific study?](https://www.lesswrong.com/posts/gxbGKa2AnQsrn3Gni/how-do-you-assess-the-quality-reliability-of-a-scientific)
    *   [Historical mathematicians exhibit a birth order effect too](https://www.lesswrong.com/posts/tj8QP2EFdP8p54z6i/historical-mathematicians-exhibit-a-birth-order-effect-too)
*   **jacobjacob**
    *   Curated posts: [The Copernican Revolution from the Inside](https://www.lesswrong.com/posts/JAAHjm4iZ2j5Exfo2/the-copernican-revolution-from-the-inside)
    *   [Unconscious Economics](https://www.lesswrong.com/posts/PrCmeuBPC4XLDQz8C/unconscious-economics)
*   **Ruby**
    *   Curated posts: [Conversation Cultures: Combat vs. Nurture](https://www.lesswrong.com/posts/ExssKjAaXEEYcnzPd/conversational-cultures-combat-vs-nurture-v2)
    *   [Why wasn't science invented in China?](https://www.lesswrong.com/posts/8SEvTvYFX2KDRZjti/answer-why-wasn-t-science-invented-in-china)

Details
-------

**When?** Sunday August 9, 1pm (PDT)

**Where?** [Zoom](https://us02web.zoom.us/j/88299438824?pwd=Rm80ZXBLMU5BSHBnUWZDRkg2NmhiQT09).
# What are some low-information priors that you find practically useful for thinking about the world?

_cross-posted on the [EA Forum](https://forum.effectivealtruism.org/posts/SBbwzovWbghLJixPn/what-are-some-low-information-priors-that-you-find)_

I'm interested in questions of the form, "I have a bit of metadata/structure to the question, but I know very little about the content of the question (or alternatively, I'm too worried about biases/hacks to how I think about the problem or what pieces of information to pay attention to). In those situations, what prior should I start with?"

I'm not sure if there is a more technical term than "low-information prior."

Some examples of what I found useful recently:

1\. Laplace's [Rule of Succession](https://en.wikipedia.org/wiki/Rule_of_succession#Statement_of_the_rule_of_succession), for when the underlying mechanism is unknown.

2\. Percentage of binary questions that resolves as "yes" on Metaculus. It turns out that of all binary (Yes-No) questions asked on the prediction platform [Metaculus](https://www.metaculus.com/questions/), 29% of them resolved yes. This means that even if you know _nothing_ about the _content_ of a Metaculus question, a reasonable starting point for answering a randomly selected binary Metaculus question is 29%.

In both cases, obviously there are reasons to override the prior (for example, you can arbitrarily flip all questions on Metaculus such that your prior is now 71%). However (I claim), having a decent prior is nonetheless useful in practice, even if it's theoretically unprincipled.

I'd be interested in seeing something like 5-10 examples of low-information priors as useful as the rule of succession or the Metaculus binary prior.
# Is Wirecutter still good?

I have heard from several friends I trust that [Wirecutter](https://www.nytimes.com/wirecutter/) is no longer very reliable since being acquired by the New York Times in 2016, and that their Wirecutter-advised purchases have become pretty mediocre in the last year or two.

I'd be interested in people making some of that case, as answers to this post, including things like talking about purchases they made or basic errors they found in the reviews, as I don't have strong, publicly verifiable evidence on this at the minute.

This is a biased post, I'm writing this with the hope of helping propagate this info if it's true, which I suspect it is but am not confident of. Please give whatever answers you feel are most relevant.
# How much is known about the "inference rules" of logical induction?

Context: [Logical Induction](https://intelligence.org/files/LogicalInduction.pdf) is a framework that makes sense of intuitively plausible statements like "the probability that the $10^{10^{10^{10}}}$th digit of $\pi$ is odd is about $0.5$".

People often do this sort of informal reasoning about mathematical conjectures. Like "The Collatz conjecture has been checked up to $2^{68}$, and held for all those - updating on this, I increase my likelyhood that the conjecture is true in general".
Logical induction seems to provide, in principle, a set of rules that such updates should follow. How many of these rules are known?

Some example rules that seem very plausible (here all my variables are implicitly natural numbers):

- The observation that $\phi(n_0)$ is true does not decrease the likelyhood of $\forall n \phi(n)$.
- Updating on the observations "$\phi(n)$ for all $0 \leq n \leq N$", the probability of $\forall n \phi(n)$ goes to $1$ as $N \to \infty$

Do these hold for logical inductors?
# 10/50/90% chance of GPT-N Transformative AI?

In [Developmental Stages of GPTs](https://www.lesswrong.com/posts/3nDR23ksSQJ98WNDm/developmental-stages-of-gpts), orthonormal explains why we might need "zero more cognitive breakthroughs" to reach transformative AI, aka "an AI capable of making an Industrial Revolution sized impact". More specifically, he says that "basically, GPT-6 or GPT-7 might do it".

Besides, [Are we in an AI overhang?](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang) makes the case that "GPT-3 is the trigger for 100x larger projects at Google, Facebook and the like, with timelines measured in months."

Now, assuming that "GPT-6 or GPT-7 might do it" and that timelines are "measured in months" for 100x larger projects, **in what year will there be a 10% (resp. 50% / 90%) chance of having transformative AI and what would the 50% timeline look like?**
# Existential Risk is a single category

A lot of people speak in terms of "existential risk from artificial intelligence" or "existential risk from nuclear war." While this is fine to some approximation, I rarely see it pointed out that this is not how risk works. Existential risk refers to the probability of a set of outcomes, and those outcomes are not defined in terms of their cause.

To illustrate why this is a problem, observe that there are numerous ways for two or more things-we-call-existential-risks to contribute equally to a bad outcome. Imagine nuclear weapons leading to a partial collapse of civilization, leading to an extremist group ending the world with an engineered virus. Do we attribute this to existential risk from nuclear weapons or from Bio-Terrorism? That question is neither well-defined, nor does it matter. All that matters is how much each factor _contributes_ to \[existential risk of any form\].

Thus, ask not "is climate change an existential risk," but "does climate change contribute to existential risk?" Everything we care about is contained in the second question.
# Forecasting AI Progress: A Research Agenda

AI forecasting is an important research area but lacking in a general direction. To address this issue we present a research agenda for AI forecasting that has been generated using the Delphi technique to elicit opinion from 15 leading researchers on the topic (the majority of whom are members of this community). The research agenda can be found on arXiv through this link:

**[Forecasting AI Progress: A Research Agenda (link to arXiv)](https://arxiv.org/abs/2008.01848)**

The agenda was framed so that it can be useful to both members of this community as well as the technological forecasting community more broadly. To these ends we plan to submit the arXiv manuscript to Technological Forecasting and Social Change, however, we will wait for roughly a month to receive comments. Please feel free to give us your thoughts here.
# Will OpenAI's work unintentionally increase existential risks related to AI?

\[The original question was "Is OpenAI increasing the existential risks related to AI?" I changed it to the current one following a [discussion](https://www.alignmentforum.org/posts/CD8gcugDu5z2Eeq7k/is-openai-increasing-the-existential-risks-related-to-ai?commentId=o5jsD7s5BfKWJYxqN) with Rohin in the comments. It clarifies that my question asks about the consequences of OpenAI's work will assuming positive and aligned intentions.\]

This is a question I've been asked recently by friends interested in AI Safety and EA. Usually this question comes from discussions around GPT-3 and the tendency of OpenAI to invest a lot in capabilities research.

\[Following this [answer](https://www.alignmentforum.org/posts/CD8gcugDu5z2Eeq7k/is-openai-increasing-the-existential-risks-related-to-ai?commentId=poyJakxpRpCt5hbEf) by Vaniver, I propose for a baseline/counterfactual the world where OpenAI doesn't exists but the researchers there still do.\]

Yet I haven't seen it discussed here. Is it a debate we failed to have, or has there already been some discussion around it? I found a [post](https://www.lesswrong.com/posts/Nqn2tkAHbejXTDKuW/openai-makes-humanity-less-safe) from 3 years ago, but I think the situation probably changed in the meantime.

A couple of arguments for and against to prompt your thinking:

*   **OpenAI is increasing the existential risks related to AI because:**

*   They are doing far more capability research than safety research;
*   They are pushing the state of the art of capability research;
*   Their results will motivate many people to go work on AI capabilities, whether out of wonder or out of fear of unemployment.

*   **OpenAI is not increasing the existential risks related to AI because:**

*   They have a top-notch safety team;
*   They restrict the access to their models, by either not releasing them outright (GPT-2) or bottlenecking access through their API (GPT-3);
*   Their results are showing the potential dangers of AI, and pushing many people to go work on AI safety.Is OpenAI increasing the existential risks related to AI?
# Money creation and debt

I'm confused about fiat money creation and debt in the US. My mental model is that the Fed lends money into existence (either to banks or to the treasury), which creates a loan (that eventually has to be paid back with interest, though I think they can choose not to collect on loans to the treasury?) (I'm also not sure where the money the Fed uses to pay interest on bank reserves comes from, but since it used to not do that I'll ignore it and pretend I'm asking about the time before that existed). The fractional reserve banking system then creates a convergent geometric series of additional money creation through repeated lending and saving, and that's basically M2. (I'm ignoring the fact that the reserve requirement is currently zero, I don't think it affects my question anyway).

  

It seems like that means that for every dollar that exists, there is an equivalent dollar owed in debt to someone. And conversely, when debt gets paid down, the monetary base decreases. This suggests that it is not possible, economy-wide, for private sector net savings to increase unless the government debt increases, and that whenever government debt goes down, private savings must decrease. I feel like I'm missing something central, because otherwise this would be public policy 101 level stuff that the Left especially would be very vocal about, but I can't figure out what.