# Information hazards: Why you should care and what you can do

_This post was written for [Convergence Analysis](https://www.convergenceanalysis.org/)._

Overview
========

We argue that many people should consider the risk that they could cause harm by developing or sharing (true) information. We think that harm from such [information hazards](https://www.lesswrong.com/posts/R7szBR5H487XutfKy/what-are-information-hazards) may sometimes be very substantial, and that this applies especially to people who research advanced technologies and/or catastrophic risks, or who often think about such technologies and risks.

However, _constantly_ worrying about information hazards would be paralyzing and unnecessary. We therefore outline a heuristic for quickly identifying whether, in a given situation, it’s worth properly thinking about the hazards some information might pose, and about how to act given those hazards. This heuristic is based on the “potency” and “counterfactual rarity” of the information in question. We next outline, and give examples to illustrate, a range of actions one could take when one _has_ identified that some information could indeed be hazardous.

Why should some people care about information hazards?
======================================================

An [information hazard](https://www.lesswrong.com/posts/R7szBR5H487XutfKy/what-are-information-hazards) is “A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm” ([Bostrom](https://nickbostrom.com/information-hazards.pdf)). There are many types of information hazards. Here are some examples [Bostrom](https://nickbostrom.com/information-hazards.pdf) gives:

> _Data hazard_: Specific data, such as the genetic sequence of a lethal pathogen or a blueprint for making a thermonuclear weapon, if disseminated, create risk.

> \[...\] _Idea hazard_: A general idea, if disseminated, creates a risk, even without a data-rich detailed specification.

> For example, the idea of using a fission reaction to create a bomb, or the idea of culturing bacteria in a growth medium with an antibiotic gradient to evolve antibiotic resistance, may be all the guidance a suitably prepared developer requires; the details can be figured out.

> \[...\] _Attention hazard_: mere drawing of attention to some particularly potent or relevant ideas or data increases risk, even when these ideas or data are already “known”.

> Because there are countless avenues for doing harm, an adversary faces a vast search task in finding out which avenue is most likely to achieve his goals. Drawing the adversary’s attention to a subset of especially potent avenues can greatly facilitate the search. For example, if we focus our concern and our discourse on the challenge of defending against viral attacks, this may signal to an adversary that viral weapons—as distinct from, say, conventional explosives or chemical weapons—constitute an especially promising domain in which to search for destructive applications.

These and other examples highlight just how large the negative impacts of true information can sometimes be - large enough to increase [global catastrophic risks](https://forum.effectivealtruism.org/posts/EMKf4Gyee7BsY2RP8/michaela-s-shortform?commentId=vgqcDG9nMdMx8c3aj), or perhaps even [existential risks](https://forum.effectivealtruism.org/posts/skPFH8LxGdKQsTkJy/clarifying-existential-risks-and-existential-catastrophes). It is worth further highlighting two underlying reasons why information could often have such large, negative impacts:

*   If an existential catastrophe occurs, it is most likely to be for [anthropogenic rather than natural reasons](https://www.fhi.ox.ac.uk/natural-vs-anthro/), and, more specifically, may be most likely to be due to the consequences of [technological developments](https://forum.effectivealtruism.org/posts/XCwNigouP88qhhei2/differential-progress-intellectual-progress-technological). Both the development of technology and the consequences technology has on the world depend in large part on the development and sharing of information. Thus, it seems plausible that, in the worst cases, information hazards could substantially increase existential risks.
*   While information _can_ be forgotten or can drift into obscurity, it isn’t really possible to “un-develop” or “un-share” information. That is, developing and/or sharing information is a fairly _[irreversible](https://www.centreforeffectivealtruism.org/blog/hard-to-reverse-decisions-destroy-option-value/)_ decision, with potentially very long-lasting and resilient impacts (see also [the unilateralist’s curse](https://forum.effectivealtruism.org/posts/EMKf4Gyee7BsY2RP8/michaela-s-shortform?commentId=y3o9YFvj4iXiAqKWa)).

Altogether, we believe that, in _some situations_ (e.g., Bostrom’s examples), developing^[By “developing”, we mean things like “coming up with” or “independently discovering through research”. Note that, with our usage of the term “develop”, it’s possible to consider the potential dangers of some information that you _might_ develop but _haven’t yet_ developed. For example, before starting research that you expect might reveal a new way to engineer a virus, you could already assess how dangerous the results of that research might be (though this assessment would of course be quite abstract and approximate).] or sharing information could have an [expected](https://concepts.effectivealtruism.org/concepts/expected-value-theory/) negative impact that is larger than the expected positive impact the developer/sharer will achieve through everything else they do in their lives. And we believe that information hazards that are simply _very large_ (rather than so large that they overwhelm the rest of one’s impact) will be much more common.

Thus, at least for some people, it seems like an important way to make their impact more positive would be to reduce the chances that they develop or share hazardous information. This could be just as important as more typical ways to “have a positive impact”.^[We would in fact further argue that there’s no relevant moral difference between “having a positive impact” and “reducing one’s negative impact”, and we hope to write about that point in the future. But that point isn’t necessary for our arguments in this post.]

But it is obviously also true that developing and sharing information is often very beneficial, and that _constantly_ worrying about information hazards would be counterproductive. Thus, in this post we will offer:

*   a heuristic for quickly identifying whether, in a given situation, it’s worth properly thinking about the hazards some information might pose, and about how to act given those hazards
*   a list of actions one could consider when one has identified that some information could indeed be hazardous

Should _you_ care about information hazards?
--------------------------------------------

But who is this relevant to? More specifically, for whom is this topic relevant _enough_ that it’s worth them spending time learning and thinking about it? We think that only a small proportion of people will _develop_ information that poses catastrophic or existential risks. However:

*   We think that there may be an unusually large proportion of such people in the effective altruist and rationalist communities.
*   We think that the _expected value_ of being aware of this topic could be high even for people who have only a small chance of developing information that poses such severe risks, given how severe those risks would be.
*   We think that a much larger proportion of people will develop information that poses at least _somewhat_ substantial risks (even if these risks aren’t catastrophic or existential).
*   It's possible to cause substantial harm merely by _sharing_ and _drawing attention to_ information that was already developed by others, and that is already known in some circles. We think that many more people will find themselves in a position to cause harm by sharing information than by initially developing information.
    *   For example, one might learn about a technology which is already mentioned in academic journals, and then try to highlight the dangers of this technology to national security policymakers who wouldn’t have read the journal articles. This could increase the chances that this technology will be weaponised, thus causing harm. (This is an _attention hazard_.)
*   Many people with malicious intent [may not be particularly intelligent or creative](https://www.lesswrong.com/posts/Ek7M3xGAoXDdQkPZQ/terrorism-tylenol-and-dangerous-information), so it may be unexpectedly easy to develop or learn about information that such people hadn’t thought of yet. Doing so could result in these people learning about this information and using it to cause harm.
*   For people and situations where it’s _not_ worth properly thinking about the hazards some information might pose, this fact can be identified using the heuristic we provide in the next section.

Ultimately, we’d say that, as a rough rule of thumb, this topic is relevant enough that it’s worth spending time learning and thinking about it for **people who research advanced technologies and/or catastrophic risks, or who often think about such technologies and risks**. We think that such people have non-negligible odds of, at some point, developing or learning of information that poses substantial risks.^[We’re also inclined to think that it would be possible to modify the basic ideas in this post so as to make them relevant to various other sets of people, and potentially to all people. But the people we’re most interested in addressing, and for whom we think _this post’s version_ of these ideas is most useful, are indeed people who research advanced technologies and/or catastrophic risks, or who often think about such technologies and risks.]

When should you think about information hazards?
================================================

We’ve argued that it’s worthwhile for people who research advanced technologies and/or catastrophic risks, or who often think about such technologies and risks, to learn and think about the topic of information hazards. But it’d be paralyzing and unnecessary for such people to _always_ worry about information hazards. So _when_, _more specifically_, should you take the time to properly think about the hazards some information might pose, and about how to act given these hazards?^[An alternative framework to ours for answering a somewhat related question is provided in [The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?](https://arxiv.org/pdf/2001.00463.pdf) (which we read after writing this post). What that article calls “Counterfactual possession” is somewhat similar to what we call “Counterfactual rarity”.]

We propose a simple heuristic to answer this question:

* If some information you may develop or share is related to advanced technologies or catastrophic risks, then think about how “**potent**” the information is.
    * If the information’s potency is _low_, **don’t worry** about the risk of information hazards; you can proceed with developing or sharing the information.
    * If the information’s potency is _high_, consider how “**counterfactually rare**” this information is.
        * If the counterfactual rarity is _low_, then consider using **“implementation-related responses**”.
        * If the counterfactual rarity is _high_, _then_ think further about how hazardous this information might be, and consider using “**information-related responses**”.
* If the information in question _isn’t_ related to advanced technologies or catastrophic risks, **don’t worry**.

We will now explain the terms and ideas in that heuristic in more detail.

The **potency** of some information depends on factors such as _how many_ people the information may affect, _how intensely_ each affected person would be affected, _how long_ the effects would last, and so on. For example, the information that it’s possible to create a nuclear weapon is much more potent than information about what you had for breakfast. (As such, there’s no need to worry about the risks posed by information about what you had for breakfast - feel free to share that with whoever might care, and then get on with your day.)

**Counterfactual rarity** essentially refers to the number of people who are likely to have already developed or learned this information (or similar information), or to develop or learn this information soon anyway. This depends on factors such as how much specialised knowledge is required to arrive at the information, how counterintuitive the information is, the incentives for developing and sharing the information, and so on.

Counterfactual rarity is important because, however potent some information is, the impact of _you_ developing or sharing that information will depend on whether the information is already widely known, and whether it’s very likely to be discovered and publicised soon in any case. For example, the information that it’s possible to create a nuclear weapon is now very widely known, so, even though the information is very potent, it’s no longer worth worrying about it _as an information hazard_.

Instead, in such cases, we should focus on what we could call **implementation-related responses**. By this, we essentially mean any actions for reducing risks _other than_ what we might call an “information-related response” (discussed below). Examples of implementation-related responses to the risks posed by nuclear weapons include trying to control access to nuclear materials and trying to establish norms against the creation of nuclear weapons. This seems more valuable (nowadays) than the _information_-related response of trying to hide the fact that nuclear weapons can be created. This is because the key bottlenecks to causing harm using nuclear weapons are now related to _implementing_ the information, rather than to _accessing_ the information in the first place.

Some information will have high potency _and_ high counterfactual rarity. This could include, for example, information about a new way to engineer a virus. _These_ are the cases in which it’s worth thinking about how hazardous the information might be.

Additionally, in such cases, you should probably _consider_ using **information-related responses**. By this we mean actions intended to prevent the development or (further) spread of potentially hazardous information, or to influence _how_ the information is developed or (further) spread. For example, you might avoid conducting a line of research that could reveal a new way to engineer a virus, or push for research of that type to require review before it's conducted, or discourage discussion of such research outside of academic publications. (Note that our purpose here is to illustrate these ideas, rather than to recommend specific actions in response to specific, actual situations.)^[One alternative, or additional, approach for deciding whether to worry about information hazards in a particular situation would be the following: Consider how similar the information under consideration is to various types of information hazards that have been described (such as by [Bostrom](https://nickbostrom.com/information-hazards.pdf) or by [Crawford et al.](https://forum.effectivealtruism.org/posts/ixeo9swGQTbYtLhji/bioinfohazards-1)), and whether this highlights a mechanism through which the information could cause harm. For example, you could recall the concept of an attention hazard, and then think about whether the article you’re considering writing could end up raising some information to the wrong people’s attention in a way that causes harm.]

The following diagram visually represents our heuristic:

![](https://res.cloudinary.com/dwbqmbkdb/image/upload/v1582489977/info_hazard_muy85k.png)

Note that both this heuristic and the response options we discuss below are intended for use at an _individual_ level, to reduce information hazards from _one’s own_ actions. We think that many of the same ideas would apply at an _organisational_ level, or if trying to analyse and reduce the information hazards _other people’s_ actions might cause, but that some modifications would have to be made.^[There are also some valuable actions one can take to build one’s _general capacity_ for avoiding or mitigating information hazards. Learning about information hazards and how to handle them is one such action. [This article](https://80000hours.org/articles/accidental-harm/#how-can-you-mitigate-these-risks) touches on some other actions for building that general capacity.]

For information with good consequences
--------------------------------------

Although it's not our focus, we should also quickly note that essentially the mirror image of this heuristic process can be followed in relation to information you suspect may have _good_ consequences.

First, consider whether the information is highly potent. If it isn’t (e.g., if the information is about how many marbles there are in a jar), then don’t bother further assessing the potential impacts of this information.

If the information _is_ highly potent, then consider whether it’s counterfactually rare. If it’s _not_ counterfactually rare, consider using implementation-related responses. For example, the information that washing one’s hands is a good idea is highly potent, but is now very widely known. As such, in many communities, trying to spread the information further may have relatively little value. However, helping people _use_ the information (e.g., by providing access to clean water) may have high value.

If the information is highly potent _and_ counterfactually rare, then it’s worth thinking about the benefits that may result from developing and sharing the information (in a mirror image of thinking about the possible harms from hazardous information). Additionally, in such cases, you should probably consider using information-related responses. For example, if the information is a cure for cancer, it may be worth considering developing the information via research, or, if the information has recently been developed, spreading the information to more doctors and patients.

What can you do about information hazards?
==========================================

Let’s say you’ve used the above heuristic, and determined that the information in question is indeed high in potency and counterfactually rarity. You know this means that you should probably think further about how hazardous this information might be, and consider using “information-related responses”. But what _specific_ responses can you employ?

We will now provide a (non-exhaustive) list of responses that may often be worth considering when dealing with potential information hazards, along with examples to illustrate each response. Note that it would sometimes be possible to combine multiple responses (i.e., they are not mutually exclusive).

These responses are very approximately ordered from the least extreme responses, worth considering when the expected harms from developing or sharing the information are relatively low, to the most extreme responses. (But we should note that, for the most part, this post won’t offer very specific guidelines on _which_ response to take in particular situations, or how to decide that. We hope to explore those questions more in future work.)

Potential responses
-------------------

**Develop and/or share the information**: The information is sufficiently potent and counterfactually rare that it was worth thinking about the risks, but you conclude that the risks are low enough (compared to the benefits) that you can go ahead and develop and/or share the information.

*   For example, an AI safety researcher may simply carry on with their research, and with communicating it as if it posed no information hazards, as the balance of risks to benefits seem acceptable to them.

**Develop the information, but don’t (yet) share it**: Similar to the above, except that you only proceed with _developing_ the information, concluding that _sharing_ it is not worthwhile, or that it’d be better to wait until later to decide whether to share it.

*   For example, an AI safety researcher may think that their research has sufficiently low risks, and sufficiently high benefits, that it’s worth _them_ proceeding with the research. However, the researcher may also think that sharing the results would be net negative, because some other people could use the results to enhance capabilities of unsafe systems. Or the researcher may decide that sharing the results _might_ be net negative, so they’ll wait to see the _full_ results before deciding whether to share anything about their research.

**Think more about the risks**: Consider in more detail how risky the information might be. You might do this while continuing to develop and/or share the information, or before doing so (with the results of your thinking determining whether to do so).

*   An AI safety researcher decides they’re too uncertain about the risks and benefits of the research they had intended to do (or about sharing the results they’ve found) to proceed with that. As such, they first try to think very carefully about those risks and benefits.

**Frame the information to reduce risks**: Make a conscious effort to frame or explain the information in a way that reduces its risks, such as by influencing how the information is likely to be used or who it is likely to reach.

*   After many interesting conversations at EA Global, an EA comes up with a new argument for why certain technologies could be very powerful and dangerous. The EA shares their argument, but emphasises how this highlights the importance of caution and [differential progress](https://forum.effectivealtruism.org/posts/XCwNigouP88qhhei2/differential-progress-intellectual-progress-technological). They also deliberately use somewhat dry, academic language, and avoid particularly dramatic or militaristic language. This is all to reduce the chances of drawing to these technologies the wrong kind of attention from the wrong kind of people.

**Develop and/or share a subset of the information**: Instead of developing and/or sharing _none_ of the information, or _all_ of it, work out what part of the information would be net-beneficial to develop and/or share, and develop and/or share that part only. (If doing this, it’s worth thinking carefully about whether the part you’re planning to develop and/or share would be sufficient for others to (re)construct the other parts of the information as well.)

*   The EA from the previous example decides to share the parts of their argument that focus on the risks of catastrophic _accidents_ from certain technologies, but not the parts that focus on the risks of catastrophic _misuse_ of those technologies. This is because the EA concludes that the recommendations they want to make can be supported without mentioning the misuse risks, and that drawing attention to the misuse risks could make such misuse more likely.

**Share the information with a subset of people**: Do share the information, but not indiscriminately. Instead, work out which people it’s likely to be net-beneficial to share the information with, and share it with just these people. (One reason to do this would be to get these people’s opinion on how dangerous it would be to further develop and/or share the information. Another would be to allow these people to develop defences against whatever harms the information relates to or could cause.)

*   An AI safety researcher shares a preliminary finding with a handful of trusted and more experienced researchers, to get advice on whether it’s wise to proceed with this line of research, and whether it’s wise to make it public.
*   A biology student discovers a weakness in the health system that could potentially be exploited by terrorists. They report this to the relevant authorities, but don’t discuss the information publicly.

**Avoid developing and/or sharing the information**: If the information poses high enough risks (relative to its potential benefits), it may be best to simply avoid developing and/or sharing it at all.

*   An AI safety researcher comes to the conclusion that a certain line of research could substantially advance capabilities of unsafe systems, without advancing safety to as great an extent. The researcher thus just drops that line of research.

**Monitor whether others may develop and/or share the information**: For sufficiently risky information, it may be worth actively investigating whether anyone else is developing or sharing the information, or whether anyone seems likely to do so in future. If you identify any such people, you might then warn them of the potential risks, let the relevant authorities know that these people may develop or share this information, and/or try to make it harder for the information to be acted on if it _is_ developed and shared (i.e., try to use implementation-related responses). You might do all this after having stopped developing and/or sharing the information yourself, or while continuing to do so (probably _very cautiously)_.

*   An AI safety researcher concludes that a certain line of research might be _very_ unsafe. The researcher decides that they should check published papers and ask around in very vague and careful ways to see if anyone else is working on a similar line of research. The researcher identifies a research group that is indeed doing so, and reaches out to that group to discuss reasons this research may be too risky to be worth pursuing.

**Decrease the likelihood of others developing and/or sharing the information:** This overlaps with parts of the above response option. But there are also ways of doing this that _don’t_ involve monitoring whether others may develop and/or share the information. For example, you could delete your own research notes, or try to steer people away from books, articles, concepts, etc. that helped lead you to the information.

*   An AI safety researcher concludes that a certain line of research might be _very_ unsafe, so they destroy all their writings related to that research, as well as their earlier writings on ideas that led them to that line of research. They also think back over what previous work and concepts from others helped inspire and lead them to that line of research, and try to avoid mentioning or recommending such work and concepts to others in future, to decrease the odds that others get inspired and led to that research in the same way.

There are also two potential responses that seem to us like they could arguably be classified either as implementation-related responses or as information-related responses. These two responses are the following:

**Develop countermeasures**: Develop ways of preventing, mitigating, or fixing the potential harms of the information. You might do all this after having stopped developing and/or sharing the information yourself, or while continuing to do so (perhaps to inform your efforts to develop countermeasures).

*   A biology student discovers a weakness in the health system that could potentially be exploited by terrorists. The student tries to think of ways to patch up that weakness, or to respond rapidly if the weakness _is_ exploited. To help with that effort, they also continue _very carefully_ researching that weakness, and contact some _very carefully_ selected people to help.

**Improve the groups that might discover or use the information**: Try to improve the values (e.g., level of altruism) and/or capabilities of the people or organisations who might develop, learn about, or implement the information. This would be done to reduce the risks that they will share or use the information in harmful ways. (See also [Improving the future by influencing actors' benevolence, intelligence, and power](https://forum.effectivealtruism.org/posts/4oGYbvcy2SRHTWgWk/improving-the-future-by-influencing-actors-benevolence).)

*   An AI policy researcher decides that it would theoretically be good for some policymakers to learn of their findings, but that that really depends on how altruistic, thoughtful, competent, and good at keeping secrets those policymakers are. They decide to enlist help in trying to improve some policymakers’ values and capabilities first, and then perhaps share the information with these policymakers later.

Again, we emphasise that this list is probably not exhaustive, that some of the response options could be used in combination, and that we aren’t here making specific suggestions on _when_ to use _which_ response. Note also that one may often be able to switch from one response option to another later (e.g., from sharing with a subset of people to sharing publicly), especially if one has started with a relatively cautious option.

Conclusion
==========

We believe that many people have a significant chance of at some point being in a position to develop and/or share information that could cause substantial harm. We think that this is especially true for people who research advanced technologies and/or catastrophic risks, or who often think about such technologies and risks. Thus, we think that responding well to potential information hazards can be a very important way for such people to have a more positive impact on the world.

We suggested a heuristic to use when one is facing a potential information hazard: first ask yourself whether the information has high potency, and then whether it has high counterfactual rarity. If the answer to both questions is no, then you don’t have to worry. If the information is highly potent but not counterfactually rare, you should consider using “implementation-related responses” (e.g., limiting access to uranium). If the information is highly potent _and_ counterfactually rare, _then_ you should think further about whether this information could be hazardous, and consider using “information-related responses”.

We then outlined a set of information-related responses one could consider, which we hope will guide readers in making more informed choices when facing potential information hazards.

There are two key things that this post has _not_ done:

*   Given clear guidelines on _which_ response options to use _in which situations_, and/or how to think through that question.
*   Clearly integrated our ideas with those developed by others (e.g., [here](https://onlinelibrary.wiley.com/doi/full/10.1111/risa.13235), [here](https://arxiv.org/pdf/2001.00463.pdf), and [here](https://forum.effectivealtruism.org/posts/ixeo9swGQTbYtLhji/bioinfohazards-1)).

We hope to explore those avenues further in future work.

_This post was written for [Convergence](https://www.convergenceanalysis.org/)_ _by MichaelA and [Justin Shovelain](https://www.lesswrong.com/users/justinshovelain), based on an earlier post written by Justin and [Andrés Gómez Emilsson](https://www.qualiaresearchinstitute.org/our-team). We’re also grateful to_ _[David Kristoffersson](https://www.lesswrong.com/users/david_kristoffersson), Aaron Gertler, and Will Bradshaw for helpful comments and edits on earlier drafts, and to Anders Sandberg and Ben Harack for helpful discussions on the general topic._