# Startup Lessons for AI Safety

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/a62ff03162c863a5b43ede03f6f2a8832d573e0dfc9788c6eb183fda73d099c9/jtnfbslxbjblrv1wt2mq)

Most ideas fail.

Successful startup founders don't get their first idea right. So they try things out, iterate relentlessly, and pivot when needed –[^wvvb66yenie] until they find something that works.

In AI Safety, most ideas won't be impactful either.

However, most people don’t know how to systematically move quickly and address the problems that need solving.

The result?

Projects that move slowly, fail to gain traction, and get stuck.

This post shows you how to avoid getting stuck in failure. We’ll introduce you to the iteration system that helps startup founders succeed.

Specifically, entrepreneurial skills help you with:

*   **Execution**: Building something . . .
*   **Adoption**: . . . that people engage with.

*This is post 2 of 5 in the sequence "How to launch an impactful AI Safety project".* [*Post 1*](https://www.lesswrong.com/posts/jDDZsoGRK8HNZEoTb/two-skillsets-you-need-to-launch-an-impactful-ai-safety) *introduces the broader topic, but this post works well as a standalone post.*

Execution: How to build something
---------------------------------

Be relentlessly resourceful.

That’s how Paul Graham – co-founder of YCombinator and legendary entrepreneurship coach – describes being a good startup founder.[^3cpv799gwk]

Beyond that, building anything requires a broad range of generalist execution skills: project and people management, finance, communication, engineering, research.

These skills are essential. You might know they matter, but it can be hard to actually acquire them. One place to start is reading the book “How to Launch an Impactful Nonprofit”.[^vpyf95toz8]

We strongly suggest you try to improve your general execution skills, or at least get feedback from people who know you well to figure out what you could work on.

Nevertheless, competent execution isn’t sufficient for success.

Most new ideas will fail, even if competently executed[^eo6we9l7at7]
--------------------------------------------------------------------

Rather than just building *something*, you have to find the*right thing* to build.

Some thing, idea, or artefact that, if you execute it well, people will actually use, talk about, or build upon. And then, additionally, all that buzz has to actually affect AI Safety outcomes.

That’s how [Impact = Adoption x Effectiveness](https://www.lesswrong.com/posts/jDDZsoGRK8HNZEoTb/two-skillsets-you-need-to-launch-an-impactful-ai-safety).

We'll discuss *adoption*in this post, and then turn to *impact* in post 3.

Adoption:  Build something people want
--------------------------------------

If people don’t want what you created, they won’t spend time with it.

To build something people actually adopt, you need to understand what they want.[^f6w1l191air] You can do this by talking to users, identifying assumptions, testing the riskiest assumptions, and iterating toward a solution.

“Users” can take many forms across AI Safety – for example: researchers building on your work; legislators implementing your policies; people joining your fellowships; or people adopting your tool.

Getting adoption is entrepreneurship 101 and there are entire books on the topic – e.g. [The Lean Startup](https://www.goodreads.com/book/show/10127019-the-lean-startup?ac=1&from_search=true&qid=H1TWmf0s0p&rank=1), [Inspired](https://www.goodreads.com/book/show/35249663-inspired?ref=nav_sb_ss_1_8), [The Mom Test](https://www.goodreads.com/book/show/52283963-the-mom-test?ref=nav_sb_ss_1_12)and [Continuous Discovery Habits](https://www.goodreads.com/book/show/58046715-continuous-discovery-habits?ref=nav_sb_ss_1_19). Podcasts like [Lenny's Podcast](https://www.lennysnewsletter.com/podcast) and entrepreneurship bootcamps teach this too.

We’ll cover just a few key takeaways in this post.

Fall in love with the problem, not the solution
-----------------------------------------------

Don’t get attached to your idea.

Don’t even get attached to a specific sub-problem within AI Safety yet.[^64fyv6chcuj] First fall in love with the entire problem of existential risk. Then identify a specific sub-problem where you have the highest expected impact.

Only then think about solutions.[^rzm1r7zb0pk]

Note that the maximum impact of your project is capped by the importance of the problem it tries to solve. If a problem causes 3% of existential risk, a solution can only reduce risk by 3%. It's often better to focus on another problem that's responsible for, e.g., 40% existential risk.[^unv9sr96ptm]

Ship fast, kill fast
--------------------

Successful entrepreneurs iterate. A lot.

They make it their goal to kill bad ideas fast, so they can get to a good idea sooner. They do that by maximizing their learning rate, and de-risking the assumptions that stand between them and success.

So go build something scrappy and get it out there. Don’t spend a lot of time on it. Optimize for learning, not for being proud of what you built. Then improve what works and scrap what doesn’t.

It’s tricky to give advice on this topic because different people need to hear different things.

*   For the talkers: stop talking and start building.
*   For the engineers: stop building and start shipping.
*   For the entrepreneurs: stop shipping and start thinking about your theory of change.[^r2ged1l7lhs]

The entrepreneurial skills from this post will help you create something that people use. But adoption alone isn't enough for impact.

Adoption ≠ Impact
-----------------

![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1774617892/lexical_client_uploads/qsmmqeeyvrnpe0fd7g2i.png)

I like bananas. If you sell them, I'll buy them. But this does approximately nothing for existential risk.

Remember: [Impact = Adoption x Effectiveness](https://www.lesswrong.com/posts/jDDZsoGRK8HNZEoTb/two-skillsets-you-need-to-launch-an-impactful-ai-safety)

Adoption without effectiveness looks like:

*   Research that gets cited but turns out to be a technical novelty that’s not useful
*   Policies that pass but don't affect existential risks
*   Fellowships that train people for jobs that don't exist
*   Or worse: new insights that help AI Safety somewhat but help AI capabilities more, bringing us to dangerous AI sooner and thus having a negative impact.

To be effective, you’ll need to know what problems need solving and what might solve them. That’s where strategy and impact assessment come in, which we'll turn to in the post 3.

Unless you're in the top ~100 AI safety experts, that’s where you’ll want to go next.

*Post 3 hasn't been published yet, but you can leave your email address below to get notified when it launches:*

```widget[oRqknDBHXk3XFrYZq]
<!-- Main React implementation: src/pages/subscribe/+Page.tsx -->
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script>
// ── CONFIG ─────────────────────────────────────────────
var CONFIG = {
  // Heading & description (set to "" to hide either)
  heading: "Don't miss the next post",
  description: "We're also looking for navigators to facilitate groups at Lens Academy: free, tutor-guided courses on AI safety, with weekly group discussions and hands-on exercises.",

  // Courses: "hide", "none", "learner", "navigator" (hide, or set none/learner/navigator as default)
  courses: "none",

  // Posts: "hide", "false", "true" (hide, or set false/true as default)
  posts: true,

  // API endpoint
  api: "https://lensacademy.org",
};
// ── END CONFIG ─────────────────────────────────────────
</script>
<link href="https://fonts.googleapis.com/css2?family=DM+Sans:wght@400;500;600;700&family=Newsreader:wght@600;700&display=swap" rel="stylesheet">
<style>
  * { margin: 0; padding: 0; box-sizing: border-box; }
  body {
    font-family: 'DM Sans', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
    background: #faf8f3;
    padding: 16px;
    display: flex;
    justify-content: center;
  }
  .card {
    background: #ffffff;
    border: 1px solid #e8e5df;
    border-radius: 12px;
    box-shadow: 0 4px 20px rgba(0,0,0,0.1);
    padding: 28px 32px;
    max-width: 600px;
    width: 100%;
  }
  h2 {
    font-family: 'Newsreader', serif;
    font-size: 26px;
    font-weight: 700;
    color: #1a1a1a;
    margin-bottom: 6px;
    line-height: 1.3;
  }
  .desc {
    font-size: 16px;
    color: #6b6b6b;
    line-height: 1.55;
    margin-bottom: 20px;
  }
  .options {
    display: flex;
    flex-direction: column;
    gap: 14px;
    margin-bottom: 18px;
  }
  .option-row {
    display: flex;
    align-items: center;
    justify-content: space-between;
    gap: 12px;
  }
  @media (max-width: 480px) {
    .option-row.courses-row {
      flex-direction: column;
      align-items: flex-start;
      gap: 6px;
    }
  }
  .option-info {
    display: flex;
    flex-direction: column;
    gap: 1px;
  }
  .option-label {
    font-size: 15px;
    color: #1a1a1a;
    font-weight: 600;
  }
  .option-sub {
    font-size: 13px;
    color: #6b6b6b;
    line-height: 1.4;
  }
  /* 3-segment pill */
  .seg-pill {
    display: inline-flex;
    background: #e8e5df;
    border-radius: 8px;
    padding: 2px;
    gap: 2px;
    flex-shrink: 0;
  }
  .seg-pill button {
    padding: 6px 14px;
    font-family: inherit;
    font-size: 14px;
    font-weight: 500;
    border: none;
    border-radius: 6px;
    background: transparent;
    color: #6b6b6b;
    cursor: pointer;
    transition: all 0.15s;
    white-space: nowrap;
  }
  .seg-pill button:hover {
    color: #1a1a1a;
    background: rgba(255,255,255,0.5);
  }
  .seg-pill button.active {
    background: #fff;
    color: #1a1a1a;
    box-shadow: 0 1px 3px rgba(0,0,0,0.1);
  }
  .seg-pill button.active-accent {
    background: #b87018;
    color: #fff;
    box-shadow: 0 1px 3px rgba(0,0,0,0.15);
  }
  /* Info tip */
  .pill-wrap {
    display: inline-flex;
    align-items: center;
    gap: 6px;
    flex-shrink: 0;
  }
  .info-tip {
    position: relative;
    display: inline-flex;
    align-items: center;
  }
  .info-tip-btn {
    display: inline-flex;
    align-items: center;
    justify-content: center;
    width: 34px;
    height: 34px;
    margin: -9px;
    background: transparent;
    border: none;
    cursor: pointer;
    padding: 0;
  }
  .info-tip-icon {
    display: inline-flex;
    align-items: center;
    justify-content: center;
    width: 18px;
    height: 18px;
    border-radius: 50%;
    border: 1.5px solid #e8e5df;
    font-size: 11px;
    font-weight: 700;
    font-style: italic;
    color: #6b6b6b;
    line-height: 1;
  }
  .info-tip-bubble {
    display: none;
    position: absolute;
    right: 0;
    bottom: 100%;
    margin-bottom: 8px;
    width: 256px;
    padding: 10px 12px;
    border-radius: 8px;
    font-size: 13px;
    line-height: 1.4;
    background: #fff;
    color: #1a1a1a;
    border: 1px solid #e8e5df;
    box-shadow: 0 8px 24px rgba(0,0,0,0.16), 0 2px 8px rgba(0,0,0,0.08);
    z-index: 50;
  }
  .info-tip-bubble.open { display: block; }
  .info-tip-bubble::before {
    content: "";
    position: absolute;
    right: 5px;
    top: 100%;
    border-left: 6px solid transparent;
    border-right: 6px solid transparent;
    border-top: 6px solid #e8e5df;
  }
  .info-tip-bubble::after {
    content: "";
    position: absolute;
    right: 5px;
    top: 100%;
    margin-top: -1px;
    border-left: 6px solid transparent;
    border-right: 6px solid transparent;
    border-top: 6px solid #fff;
  }
  /* Toggle switch */
  .toggle {
    position: relative;
    width: 40px;
    height: 22px;
    flex-shrink: 0;
  }
  .toggle input {
    opacity: 0;
    width: 0;
    height: 0;
  }
  .toggle-track {
    position: absolute;
    inset: 0;
    background: #ccc;
    border-radius: 11px;
    cursor: pointer;
    transition: background 0.2s;
  }
  .toggle-track::after {
    content: "";
    position: absolute;
    top: 2px;
    left: 2px;
    width: 18px;
    height: 18px;
    background: #fff;
    border-radius: 50%;
    transition: transform 0.2s;
    box-shadow: 0 1px 2px rgba(0,0,0,0.15);
  }
  .toggle input:checked + .toggle-track {
    background: #b87018;
  }
  .toggle input:checked + .toggle-track::after {
    transform: translateX(18px);
  }
  .form-row {
    display: flex;
    gap: 8px;
  }
  input[type="email"] {
    flex: 1;
    min-width: 0;
    padding: 10px 14px;
    font-family: inherit;
    font-size: 16px;
    border: 1px solid #e8e5df;
    border-radius: 8px;
    outline: none;
    background: #fff;
    color: #1a1a1a;
  }
  input[type="email"]:focus {
    border-color: #b87018;
    box-shadow: 0 0 0 2px rgba(184, 112, 24, 0.15);
  }
  input[type="email"]::placeholder {
    color: #999;
  }
  .btn-submit {
    padding: 10px 22px;
    font-family: inherit;
    font-size: 16px;
    font-weight: 600;
    border: none;
    border-radius: 8px;
    background: #b87018;
    color: #fff;
    cursor: pointer;
    white-space: nowrap;
    transition: background 0.15s;
  }
  .btn-submit:hover { background: #9a5c10; }
  .btn-submit:disabled { opacity: 0.6; cursor: default; }
  .success {
    font-size: 16px;
    color: #b87018;
    font-weight: 600;
    padding: 8px 0;
  }
  .substack-note {
    font-size: 14px;
    font-weight: 400;
    color: #6b6b6b;
    margin-top: 6px;
  }
  .error {
    font-size: 14px;
    color: #c0392b;
    margin-top: 6px;
  }
</style>
</head>
<body>
<div class="card">
  <h2 id="heading">Lens Academy</h2>
  <p class="desc" id="desc">
    Free, tutor-guided courses on AI safety, with weekly group discussions
    and hands-on exercises.
  </p>

  <div id="form-container">
    <div class="options">
      <div class="option-row courses-row" id="courses-row">
        <div class="option-info">
          <span class="option-label">Courses</span>
          <span class="option-sub">Get notified when courses open for enrollment</span>
        </div>
        <div class="pill-wrap">
          <div class="seg-pill" id="course-pill">
            <button type="button" data-val="none" onclick="setCourse('none')">None</button>
            <button type="button" data-val="learner" class="active-accent" onclick="setCourse('learner')">Learner</button>
            <button type="button" data-val="navigator" onclick="setCourse('navigator')">Navigator</button>
          </div>
          <span class="info-tip" id="info-tip">
            <button type="button" class="info-tip-btn" aria-label="More info" onclick="toggleInfo()">
              <span class="info-tip-icon">i</span>
            </button>
            <div class="info-tip-bubble" id="info-bubble">Navigators facilitate the weekly group calls, guiding discussion and helping learners stay on track.</div>
          </span>
        </div>
      </div>
      <div class="option-row" id="posts-row">
        <div class="option-info">
          <span class="option-label">Posts</span>
          <span class="option-sub">Get notified when Luc or Lens Academy publishes a new post</span>
        </div>
        <label class="toggle">
          <input type="checkbox" id="cb-substack" checked />
          <span class="toggle-track"></span>
        </label>
      </div>
    </div>
    <div class="form-row">
      <input type="email" id="email" placeholder="you@example.com" />
      <button class="btn-submit" id="btn" onclick="submit()">Notify Me</button>
    </div>
    <div id="error" class="error" style="display:none;"></div>
  </div>
  <div id="success" class="success" style="display:none;">
    You're all set! We'll keep you in the loop.
    <p id="substack-note" class="substack-note" style="display:none;">
      You'll receive a confirmation email from Substack in 1–6 hours — please confirm it to start receiving posts. There's a delay in this, so there's no need to check your email right now.
    </p>
  </div>

</div>

<script>
  var courseMode = CONFIG.courses || "none";

  function toggleInfo() {
    var bubble = document.getElementById("info-bubble");
    bubble.classList.toggle("open");
  }
  (function() {
    var tip = document.getElementById("info-tip");
    tip.addEventListener("mouseenter", function() {
      document.getElementById("info-bubble").classList.add("open");
    });
    tip.addEventListener("mouseleave", function() {
      document.getElementById("info-bubble").classList.remove("open");
    });
  })();

  function setCourse(val) {
    courseMode = val;
    var btns = document.querySelectorAll("#course-pill button");
    btns.forEach(function(b) {
      b.className = b.getAttribute("data-val") === val
        ? (val === "none" ? "active" : "active-accent")
        : "";
    });
  }

  async function submit() {
    var email = document.getElementById("email").value.trim();
    var btn = document.getElementById("btn");
    var err = document.getElementById("error");
    var subscribeSubstack = document.getElementById("cb-substack").checked;
    err.style.display = "none";

    if (courseMode === "none" && !subscribeSubstack) {
      err.textContent = "Please select at least one option.";
      err.style.display = "block";
      return;
    }

    if (!email || !email.match(/^[^@\s]+@[^@\s]+\.[^@\s]+$/)) {
      err.textContent = "Please enter a valid email address.";
      err.style.display = "block";
      return;
    }

    btn.disabled = true;
    btn.textContent = "...";

    try {
      var resp = await fetch(CONFIG.api + "/api/subscribe", {
        method: "POST",
        mode: "cors",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          email: email,
          subscribe_courses_learners: courseMode === "learner",
          subscribe_courses_navigators: courseMode === "navigator",
          subscribe_substack: subscribeSubstack
        })
      });
      var data = await resp.json();
      if (!resp.ok) {
        err.textContent = data.detail || "Something went wrong.";
        err.style.display = "block";
        btn.disabled = false;
        btn.textContent = "Notify Me";
        return;
      }

      document.getElementById("form-container").style.display = "none";
      document.getElementById("success").style.display = "block";
      if (subscribeSubstack) {
        document.getElementById("substack-note").style.display = "block";
      }
    } catch(e) {
      err.textContent = "Connection error. Please try again.";
      err.style.display = "block";
    }
    btn.disabled = false;
    btn.textContent = "Notify Me";
  }

  document.getElementById("email").addEventListener("keydown", function(e) {
    if (e.key === "Enter") submit();
  });

  // Apply CONFIG
  (function init() {
    var heading = document.getElementById("heading");
    var desc = document.getElementById("desc");
    var coursesRow = document.getElementById("courses-row");
    var postsRow = document.getElementById("posts-row");

    if (!CONFIG.heading) heading.style.display = "none";
    else heading.textContent = CONFIG.heading;

    if (!CONFIG.description) desc.style.display = "none";
    else desc.textContent = CONFIG.description;

    if (CONFIG.courses === "hide") {
      coursesRow.style.display = "none";
      courseMode = "none";
    } else {
      setCourse(CONFIG.courses);
    }

    if (CONFIG.posts === "hide") {
      postsRow.style.display = "none";
      document.getElementById("cb-substack").checked = false;
    } else {
      document.getElementById("cb-substack").checked = !!CONFIG.posts;
    }
  })();
</script>
</body>
</html>

```

*(more info about Lens Academy:* [*Superintelligence Risk Education that Scales*](https://www.lesswrong.com/posts/Lg3tZCXC8NMGbFrzm/superintelligence-risk-education-that-scales-lens-academy)*)*

[^wvvb66yenie]: Emdashes; don’t you love them? This post was written and rewritten several times by humans. We did also take suggestions from AI. Nevertheless, most of the emdashes in this article are actually from humans; it’s one of the things we’ve learned from AI as they’re actually linguistically nice. Fun fact: we actually use endashes with spaces (foo – bar) instead of emdashes (foo—bar) because for some reason the main author (Luc) likes them better visually and because they smell a little less like AI while still giving the linguistic benefits. Fyi, there’s the dash (-), endash (–), and the emdash (—). 

[^3cpv799gwk]: Introduced here: https://www.paulgraham.com/relres.html. And somewhat actually explained here: https://www.jasonshen.com/how-to-be-relentlessly-resourceful/ (though this is just one interpretation) 

[^vpyf95toz8]: You can get the book at https://www.amazon.com/Launch-High-Impact-Nonprofit-Joey-Savoie-ebook/dp/B09NP91L31/ . Sadly, this book is paywalled. 

[^eo6we9l7at7]: In his book "The Right It", Alberto Savoia refers to this as "the law of market failure". 

[^f6w1l191air]: In startup jargon, this is often called 'value' or 'desirability'. There are other factors to consider, like 'usability', business 'feasibility', and having a working 'growth engine'. However, ‘value’ ("Do users actually want your product?”) is often the most important. 

[^64fyv6chcuj]: Examples of sub-problems: ASI x-risk, AIxBio, Power Concentration, Gradual Disempowerment. Don’t get attached to any of these. 

[^rzm1r7zb0pk]: This process of understanding the problem first before thinking about the solution is described by e.g. the "Double Diamond" process. Of course, progress in reality is usually not so linear. But it's a useful heuristic nonetheless. 

[^unv9sr96ptm]: Focusing on an important problem is a useful heuristic, but of course tractability and neglectedness matter too. In the end, it's about the project's expected percentage reduction of existential risk, which is based on some combination of problem, solution, and counterfactuals. 

[^r2ged1l7lhs]: Your theory of change helps your project be impactful, instead of just making you rich. We’ll discuss it in post 3.