# FAI Research Constraints and AGI Side Effects

**Ozzie Gooen and Justin Shovelain**

Summary
-------

Friendly artificial intelligence (FAI) researchers have at least two significant challenges. First, they must produce a significant amount of FAI research in a short amount of time. Second, they must do so without producing enough general artificial intelligence (AGI) research to result in the creation of an unfriendly artificial intelligence (UFAI). We estimate the requirements of both of these challenges using two simple models.

Our first model describes a _friendliness ratio_ and a _leakage ratio_ for _FAI research projects_. These provide limits on the allowable amount of artificial general intelligence (AGI) knowledge produced per unit of FAI knowledge in order for a project to be net beneficial.

Our second model studies a hypothetical _FAI venture_, which is responsible for ensuring FAI creation. We estimate necessary total FAI research per year from the venture and leakage ratio of that research. This model demonstrates a trade off between the speed of FAI research and the proportion of AGI research that can be revealed as part of it. If FAI research takes too long, then the acceptable leakage ratio may become so low that it would become nearly impossible to safely produce any new research.

Introduction
------------

A general artificial intelligence (AGI) is an AI that could perform all the intellectual tasks a human can.\[1\] When one is created, it may recursively become more intelligent to the point where it is vastly superior to human intelligences.\[2\] This AGI could be either friendly or unfriendly, where friendliness means it would have values that humans would favor, and unfriendliness means that it would not.\[3\]

It is likely that if we do not explicitly understand how to make a friendly general artificial intelligence, then by the time we make a general artificial intelligence, it will be unfriendly.\[4\] It is also likely that we are much further from understanding how to make a friendly artificial intelligence than we are from understanding how to make a general artificial intelligence.\[5\]\[6\]

Thus, it is important to create more FAI research, but it may also be important to make sure to not produce much AGI research when doing so. If it is 10 times as difficult to understand how to make an FAI than to understand how to make an AGI, then a FAI research paper that produces 0.3 equivalent papers worth of AGI research will probably increase the chances of a UFAI. Given the close relationship of FAI and AGI research, producing FAI research with a net positive impact may be difficult to do.

Model 1. The Friendliness and Leakage Ratios for an FAI Project
---------------------------------------------------------------

### The Friendliness Ratio

Let's imagine that there is necessary amount of research to build an AGI, _G~remaining~_. There is also some necessary amount of research to build a FAI, _F~remaining~_. These two have units of _r~general~_ (general AI research) and _r~friendly~_ (friendly AI research), which are not precisely defined but are directly comparable.

Which threshold is higher? According to much of the research in this field, _F~remaining~_. We need significantly more research to create a friendly AI than an unfriendly one.

![](http://www.convergenceanalysis.org/wp-content/uploads/2020/07/AGI_FAI_ratio_columns.png)

#### Figure 1. Example research thresholds for AGI and FAI.

To understand the relationship between these thresholds, we use the following equation.

[![](http://latex.codecogs.com/gif.latex?\small&space;f_{global}=\frac{\mathit{F_{remaining}}}{\mathit{G_{remaining}}} "\small f_{global}=\frac{\mathit{F_{remaining}}}{\mathit{G_{remaining}}}")](http://www.codecogs.com/eqnedit.php?latex=\small&space;f_{global}=\frac{\mathit{F_{remaining}}}{\mathit{G_{remaining}}})

We call this the **friendliness ratio**. The friendliness ratio is useful for a high level understanding of world total FAI research requirements and is a heuristic guide for how difficult the problem of differential technological development is.

The friendliness ratio would be high if _F~remaining~_ \> _G~remaining~_. For example, if there are 2000 units of remaining research for an FAI and 20 units for an AGI, the friendliness ratio would be 100. If someone published research with 20 units of FAI research but 1 unit of AI research, their research would not meet the friendliness ratio requirement (100 vs 20/1) and would thus make the problem even worse.

### The Leakage Ratio

For specific projects it may be useful to have a measure that focuses directly on the negative outcome.

For this we can use the **leakage ratio**, which represents the amount of undesired AGI research created per unit of FAI research. It is simply the inverse of the friendliness ratio.

[![](http://latex.codecogs.com/gif.latex?\small&space;l_{global}=\frac{\mathit{G_{threshold}}}{\mathit{F_{remaining}}} "\small l_{global}=\frac{\mathit{G_{threshold}}}{\mathit{F_{remaining}}}")](http://www.codecogs.com/eqnedit.php?latex=\small&space;l_{global}=\frac{\mathit{G_{threshold}}}{\mathit{F_{remaining}}})

[![](http://latex.codecogs.com/gif.latex?\small&space;l_{project}=\frac{\mathit{G_{project}}}{\mathit{F_{project}}} "\small l_{project}=\frac{\mathit{G_{project}}}{\mathit{F_{project}}}")](http://www.codecogs.com/eqnedit.php?latex=\small&space;l_{project}=\frac{\mathit{G_{project}}}{\mathit{F_{project}}})

In order for a project to be net beneficial,

[![](http://latex.codecogs.com/gif.latex?l_{project}<l_{global} "l_{project}<l_{global}")](http://www.codecogs.com/eqnedit.php?latex=l_{project}<l_{global})

###   

### Estimating if a Project is Net Friendliness Positive

#### Question: How can one estimate if a project is net friendliness-positive?

A **naive** answer would be to make sure that it falls over the global _friendliness ratio_ or under the global _leakage ratio_.

Global AI research rates need to fulfill the friendliness ratio in order to produce a FAI. Therefore, if an advance in friendliness research gets produced with FAI research _F~project~_, but in the process it also produces AGI research _G~project~_, then this would be net friendliness negative if 

[![](http://latex.codecogs.com/gif.latex?\frac{F_{project}}{G_{project}}<f_{global} "\frac{F_{project}}{G_{project}}<f_{global}")](http://www.codecogs.com/eqnedit.php?latex=\frac{F_{project}}{G_{project}}<f_{global})

Later research would need to make up for this under-balance.

### AI Research Example

Say that _G~remaining~_ = 200_r~g~_ and _F~remaining~_ =2000_r~f~_, leading to a friendliness ratio of _f~global~_ = 10 and a global maximum leakage ratio of _l~global~_ = 0.1. In this case, specific research projects could be evaluated to make sure that they meet this threshold. One could imagine an organization deciding what research to do or publish using the following chart.

|     | Description | AGI Research [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;G_{p} "G_{p}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;G_{p}) | FAI Research [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;F_{p} "F_{p}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;F_{p}) | Friendliness Ratio [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;f_{p} "f_{p}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;f_{p}) | Leakage Ratio [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;l_{p} "l_{p}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;l_{p}) |
| --- | --- | --- | --- | --- | --- |
| Project 1 | Rat simulation | [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;10r_{g} "10r_{g}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;10r_{g}) | [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;60r_{f} "60r_{f}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;60r_{f}) | 6   | 0.17 |
| Project 2 | Math Paper | [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;2r_{g} "2r_{g}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;2r_{g}) | [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;22r_{f} "22r_{f}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;22r_{f}) | 11  | 0.09 |
| Project 3 | Technical FAI Advocacy | [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;1r_{g} "1r_{g}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;1r_{g}) | [![](http://latex.codecogs.com/gif.latex?\\inline&space;\\dpi{100}&space;14r_{f} "14r_{f}")](http://www.codecogs.com/eqnedit.php?latex=\\inline&space;\\dpi{100}&space;14r_{f}) | 14  | 0.07 |

In this case, only Projects 2 and 3 have a leakage ratio of less than 0.1, meaning that only these would net beneficial. Even though Project 1 has generated safety research, it would be net negative.

### Model 1 Assumptions:

1\. There exists some threshold _G~remaining~_ of research necessary to generate an unfriendly artificial intelligence.

2\. There exists some threshold _F~remaining~_ of research necessary to generate a friendly artificial intelligence.

3\. If _G~remaining~_ is reached before _F~remaining~_, a UFAI will be created. If after, an FAI will be created.

  

---

  

---

Model 2. AGI Leakage Limits of an FAI Venture
---------------------------------------------

**Question: How can an FAI venture ensure the creation of an FAI?**

Let's imagine a group that plans to ensure that an FAI is created. We call this an FAI Venture.

This venture would be constrained by time. AGI research is being created internationally and, if left alone, will likely create an UFAI. We can consider research done outside of the venture as _external research_ and research within the venture as _internal research_. If internal research is done too slowly, or if it leaks too much AGI research, an unfriendly artificial intelligence could be created before _F~remaining~_ is met.

We thus split up friendly and unfriendly research creation into two categories, _external_ and _internal_ research. Then we consider the derivative of each with respect to time. For simplicity, we assume the unit of time is years.

_G'~i~_ = AGI research produced internally per year

_F'~i~_ = FAI research produced internally per year

_G'~e~_ = AGI research produced externally per year

_F'~e~_ = FAI research produced externally per year

We can understand that there exists times, _t~f~_ and _t~g~_, which are the times at which the friendly and general remaining thresholds are met.

_t~f~_ = Year in which _F~remaining~_ is met

_t~g~_ = Year in which _G~remaining~_ is met

These times can be estimated as follows:

[![](http://latex.codecogs.com/gif.latex?\frac{G_{remaining}}{G'_{i}+G'_{e}}=t_{g} "\frac{G_{remaining}}{G'_{i}+G'_{e}}=t_{g}")](http://www.codecogs.com/eqnedit.php?latex=\frac{G_{remaining}}{G'_{i}+G'_{e}}=t_{g})

[![](http://latex.codecogs.com/gif.latex?\frac{F_{remaining}}{F'_{i}+F'_{e}}=t_{f} "\frac{F_{remaining}}{F'_{i}+F'_{e}}=t_{f}")](http://www.codecogs.com/eqnedit.php?latex=\frac{F_{remaining}}{F'_{i}+F'_{e}}=t_{f})

The venture wants to make sure that _t~f ~_< _t~g~_ so that the eventual AI is friendly (assumption 3). With this, we find that:

[![](http://latex.codecogs.com/gif.latex?\inline&space;\small&space;F'_{i}>C_{0}+C_{1}\cdot&space;G'_{i} "\small F'_{i}>C_{0}+C_{1}\cdot G'_{i}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\small&space;F'_{i}>C_{0}+C_{1}\cdot&space;G'_{i})

Where the values of _C~0~_ and _C~1~_ both include the friendliness ratio [![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{100}&space;f_{global}=\frac{F_{remaining}}{G_{remaining}} "f_{global}=\frac{F_{remaining}}{G_{remaining}}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{100}&space;f_{global}=\frac{F_{remaining}}{G_{remaining}}).

[![](http://latex.codecogs.com/gif.latex?\inline&space;\small&space;C_{0}=f_{global}G'_{e}-F'_{e} "\small C_{0}=f_{global}G'_{e}-F'_{e}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\small&space;C_{0}=f_{global}G'_{e}-F'_{e})

[![](http://latex.codecogs.com/gif.latex?\inline&space;\small&space;C_{1}=f_{global} "\small C_{1}=f_{global}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\small&space;C_{1}=f_{global})

This implies a linear relationship between _F'~i~_ and _G'~i~_. The more FAI research the FAI venture can produce, the more AGI research it is allowed to leak.

This gives us a clean way to go from a _G'~i~_ value the venture could expect to the _F'~i~_ it would need to be successful.

The _C~0~_ value describes the absolute minimum amount of FAI research necessary in order to have a chance at a successful outcome. While the resulting acceptable leakage ratio at this point would be impossible to meet, the baseline is easy to calculate. Assuming that _F'~e~_ << _f~global~G'~e~_, we can estimate that 

[![](http://latex.codecogs.com/gif.latex?\inline&space;\small&space;C_{0}\approx&space;f_{global}G'_{e} "\small C_{0}\approx f_{global}G'_{e}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\small&space;C_{0}\approx&space;f_{global}G'_{e})

If we wanted to instead calculate _G'~i~_ using _F'~i~_, we could use the following equation. This may be more direct to the intentions of a venture (finding the acceptable amount of AGI leakage after estimating FAI productivity).

[![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{120}&space;\small&space;G'_{i}<-(G'_{e}+\frac{F'_{e}}{f_{global}})+(\frac{1}{f_{global}})\cdot&space;F'_{i} "\small G'_{i}<-(\frac{G'_{e}+F'_{e}}{f_{global}})+(\frac{1}{f_{global}})\cdot F'_{i}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{120}&space;\small&space;G'_{i}<-(G'_{e}+\frac{F'_{e}}{f_{global}})+(\frac{1}{f_{global}})\cdot&space;F'_{i})

###   

### Model 2 Example

For example, let's imagine that the [![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{100}&space;f_{global}=10\frac{r_{f}}{r_{g}} "f_{global}=10\frac{r_{f}}{r_{g}}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{100}&space;f_{global}=10\frac{r_{f}}{r_{g}}) and [![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{100}&space;G'_{e}=10\frac{r_{g}}{year} "G'_{i}=10\frac{r_{g}}{year}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{100}&space;G'_{i}=10\frac{r_{g}}{year}). In this case, [![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{100}&space;C_{0}=100\frac{r_{f}}{year} "C_{0}=100\frac{r_{f}}{year}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{100}&space;C_{0}=100\frac{r_{f}}{year}). This means that if the venture could make sure to leak exactly [![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{100}&space;0\frac{r_{g}}{year} "0\frac{r_{g}}{year}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{100}&space;0\frac{r_{g}}{year}), it would need to average a FAI research rate of 10 times that of the entire world's output of AGI research. This amount increases as 100 / (1 - 10 * l~project~). If the venture expects an estimated leakage ratio of 0.05, they would need to double their research output to [![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{100}&space;C_{0}=200\frac{r_{f}}{year} "C_{0}=100\frac{r_{f}}{year}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{100}&space;C_{0}=100\frac{r_{f}}{year}), or 20 times global AGI output.

![](http://www.convergenceanalysis.org/wp-content/uploads/2020/07/AGI_rate_vs_FAI_rate_graph.png)

Figure 2. _F'~i~_ per unit of maximum permissible _G'~i~_

#### What to do?

The numbers in the example above are a bit depressing. There is so much global AI research that it seems difficult to imagine the world averaging an even higher rate of FAI research, which would be necessary if the friendliness ratio is greater than 1.

There are some upsides. First, much hard AI work is done privately in technology companies without being published, limiting _G'~i~_. Second, the numbers of _r~g~_ and _r~f~_ don't perfectly correlate with the difficulty to reach them. It may be that we have diminishing marginal returns with our current levels of _r~g~_, so similar levels of _r~f~_ will be easier to reach.

It's possible that _F~remaining~_ may be surprisingly low or that _G~remaining~_ may be surprisingly high.

Projects with high leakage ratios don't have to be completely avoided or hidden. The _G'~i~_ value is specifically for research that will be in the hands of the group that eventually creates a AGI, so it would make sense that FAI research organizations could share high risk information between each other as long as it doesn't leak externally. The FAI venture mentioned above could be viewed as a collection of organizations rather than one specific one. It may even be difficult for AGI research implications to move externally, if the FAI academic literature is significantly separated from AGI academic literature. This logic provides a heuristic guide to choosing research projects, choosing if to publish research already done, and managing concentrations of information.

### Model 2 Assumptions:

1-3. The same 3 assumptions for the previous model.

4\. The rates of research creation will be fairly constant.

5\. External and internal rates of research do not influence each other.

Conclusion
----------

The _friendliness_ _ratio_ provides a high-level understanding of the amount of global FAI research per unit AGI research needed to create an FAI. The _leakage ratio_ is the inverse of the friendliness ratio applied to a specific _FAI project_, to specify if that specific project is net friendliness positive. These can be used to understand the challenge for AGI research and tell if a particular project is net beneficial or net harmful.

To understand the challenges facing an _FAI Venture_, we found the simple equation

[![](http://latex.codecogs.com/gif.latex?\inline&space;\small&space;F'_{i}>C_{0}+C_{1}\cdot&space;G'_{i} "\small F'_{i}>C_{0}+C_{1}\cdot G'_{i}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\small&space;F'_{i}>C_{0}+C_{1}\cdot&space;G'_{i})

where

[![](http://latex.codecogs.com/gif.latex?\inline&space;\small&space;C_{0}=f_{global}G'_{e}=F'_{e} "\small C_{0}=f_{global}G'_{e}=F'_{e}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\small&space;C_{0}=f_{global}G'_{e}=F'_{e})

[![](http://latex.codecogs.com/gif.latex?\inline&space;\small&space;C_{1}=f_{global} "\small C_{1}=f_{global}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\small&space;C_{1}=f_{global})

This paper was focused on establishing the mentioned models instead of estimating input values. If the models are considered useful, there should be more research to estimate these numbers. The models could also be improved to incorporate uncertainty, the growing returns of research, and other important limitations that we haven't considered. Finally, the friendliness ratio concept naturally generalizes to other technology induced existential risks.

Appendix
--------

a. Math manipulation for Model 2

[![](http://latex.codecogs.com/gif.latex?\frac{G_{remaining}}{G'_{i}+G'_{e}}=t_{g} "\frac{G_{remaining}}{G'_{i}+G'_{e}}=t_{g}")](http://www.codecogs.com/eqnedit.php?latex=\frac{G_{remaining}}{G'_{i}+G'_{e}}=t_{g})

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;\frac{F_{remaining}}{F'_{i}+F'_{e}}=t_{f} "\frac{F_{remaining}}{F'_{i}+F'_{e}}=t_{f}")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;\frac{F_{remaining}}{F'_{i}+F'_{e}}=t_{f})

[![](http://latex.codecogs.com/gif.latex?t_{f}<t_{g} "t_{f}<t_{g}")](http://www.codecogs.com/eqnedit.php?latex=t_{f}<t_{g})

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;\frac{F_{remaining}}{F'_{i}+F'_{e}}<\frac{G_{remaining}}{G'_{i}+G'_{e}} "\frac{F_{remaining}}{F'_{i}+F'_{e}}<\frac{G_{remaining}}{G'_{i}+G'_{e}}")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;\frac{F_{remaining}}{F'_{i}+F'_{e}}<\frac{G_{remaining}}{G'_{i}+G'_{e}})

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;G'_{i}<(\frac{G_{remaining}\cdot F'_{e}}{F_{remaining}}-G'_{e})(\frac{G_{remaining}}{F_{remaining}}F'_{i}) "G'_{i}<(\frac{G_{remaining}\cdot F'_{e}}{F_{remaining}}-G'_{e})(\frac{G_{remaining}}{F_{remaining}}F'_{i})")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;G'_{i}<(\frac{G_{remaining}\cdot F'_{e}}{F_{remaining}}-G'_{e})(\frac{G_{remaining}}{F_{remaining}}F'_{i}))

[  
![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;F_{i}>(\frac{F_{remaining}\cdot G'_{e}}{G_{remaining}}-F'_{e})(\frac{F_{remaining}\cdot G'_{i}}{G_{remaining}}) "F_{i}>(\frac{F_{remaining}\cdot G'_{e}}{G_{remaining}}-F'_{e})(\frac{F_{remaining}\cdot G'_{i}}{G_{remaining}})")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;F_{i}>(\frac{F_{remaining}\cdot G'_{e}}{G_{remaining}}-F'_{e})(\frac{F_{remaining}\cdot G'_{i}}{G_{remaining}}))

  

This last equation can be written as

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;F'_{i}>C_{0}+C_{1}\cdot&space;G'_{i} "F'_{i}>C_{0}+C_{1}\cdot G'_{i}")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;F'_{i}>C_{0}+C_{1}\cdot&space;G'_{i})

Where

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;C_{0}=\frac{F_{remaining}\cdot&space;G'_{e}}{G_{remaining}}-F'_{e} "C_{0}=\frac{F_{remaining}\cdot G'_{e}}{G_{remaining}}-F'_{e}")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;C_{0}=\frac{F_{remaining}\cdot&space;G'_{e}}{G_{remaining}}-F'_{e})

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;C_{1}=\frac{F_{remaining}}{G_{remaining}} "C_{1}=\frac{F_{remaining}}{G_{remaining}}")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;C_{1}=\frac{F_{remaining}}{G_{remaining}})

Recalling the friendliness ratio, [![](http://latex.codecogs.com/gif.latex?\inline&space;\dpi{100}&space;f_{global}=\frac{F_{remaining}}{G_{remaining}} "f_{global}=\frac{F_{remaining}}{G_{remaining}}")](http://www.codecogs.com/eqnedit.php?latex=\inline&space;\dpi{100}&space;f_{global}=\frac{F_{remaining}}{G_{remaining}}), we can simplify these constructs further.

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;C_{0}=f_{global}G'_{e}-F'{e} "C_{0}=f_{global}G'_{e}-F'{e}")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;C_{0}=f_{global}G'_{e}-F'{e})

[![](http://latex.codecogs.com/gif.latex?\dpi{100}&space;C_{1}=f_{global} "C_{1}=f_{global}")](http://www.codecogs.com/eqnedit.php?latex=\dpi{100}&space;C_{1}=f_{global})

References
----------

\[1\] What is AGI? [https://intelligence.org/2013/08/11/what-is-agi/](https://intelligence.org/2013/08/11/what-is-agi/), 2013, Luke Muehlhauser

\[2\] Intelligence Explosion FAQ, ([https://intelligence.org/ie-faq/](https://intelligence.org/ie-faq/)), MIRI

\[3\] Artificial Intelligence as a Positive and Negative Factor in Global Risk, 2008, Global Catastrophic Risks, Yudkowsky

\[4\] Aligning Superintelligence with Human Interest: A Technical Research Agenda, [https://intelligence.org/files/TechnicalAgenda.pdf](https://intelligence.org/files/TechnicalAgenda.pdf), Nate Soares and Benja Fellenstein, MIRI

\[5\] Superintelligence, 2014, Nick Bostrom

\[6\] The Challengeof Friendly AI, [https//www.youtube.com/watch?v=nkB1e-JCgmY&noredirect=1](/https/www.youtube.com/watch?v=nkB1e-JCgmY&noredirect=1) Yudkowsky, 2007