# Questions about AI that bother me 

*Crossposted from the EA Forum:* [*https://forum.effectivealtruism.org/posts/4TcaBNu7EmEukjGoc/questions-about-ai-that-have-bothered-me*](https://forum.effectivealtruism.org/posts/4TcaBNu7EmEukjGoc/questions-about-ai-that-have-bothered-me)

As 2022 comes to an end, I thought it'd be good to maintain a list of "questions that bother me" in thinking about AI safety and alignment. I don't claim I'm the first or only one to have thought about them. I'll keep updating this list.

(The title of this post alludes to the book "[Things That Bother Me](https://www.penguinrandomhouse.com/books/563023/things-that-bother-me-by-galen-strawson/)" by Galen Strawson)

First posted: 12/6/22

Last updated: 1/30/23

### General Cognition

*    What signs do I need to look for to tell whether a model's cognition has started to emerge, e.g., situational awareness?
*   Will a capacity for "doing science" be sufficient condition for general intelligence?
*   How easy was it for humans to get science (e.g., compared to evolving to take over the world). 

### Deception 

*    What kind of interpretability tools do we need to avoid deception? 
*    How do we get these interpretability tools and even if we do get them, what if they're like neuroscience for understanding brains (not enough)?
*    How can I tell whether a model has found another goal to optimize for during its training?
*    What is it that makes a model switch to a goal different from the one set by the designer? How do you prevent it from doing so?

### Agent Foundations 

*   Is the description/modeling of an agent ultimately a mathematical task?
*   From where do human agents derive their goals?
*   [Is value fragile](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile)? 

### Theory of Machine Learning

*   What explains the success of deep neural networks?
*   Why was connectionism unlikely to succeed? 

### Epistemology of Alignment (I've written about this [here](https://forum.effectivealtruism.org/s/sC8KoZx9jAdrEtmHj))

*   How can we accelerate research?
*   Has philosophy ever really helped scientific research e.g., with concept clarification?
*   What are some concrete takeaways from the history of science and technology that could be used as advice for alignment researchers and field-builders? 
*   [The emergence of the AI Safety paradigm](https://forum.effectivealtruism.org/s/sC8KoZx9jAdrEtmHj/p/pC9RJdmP3rnhuHpCm)

### Philosophy of Existential Risk 

*   What is the best way to explain the difference between forecasting extinction scenarios and narratives from chiliasm or escatology? 
*   What is the best way to think about serious risks in the future without reinforcing a sense of doom? 

### Teaching and Communication

*   Younger people (e.g., my undergraduate students) seem more willing to entertain scenarios of catastrophes and extinction compared to older people (e.g., academics). I find that strange and I don't have a good explanation as to why that is the case. 
*   The idea of a technological singularity was not difficult to explain and discuss with my students. I think that's surprising given how powerful the weirdness heuristic is.
*   The idea of "agency" or "being an agent" was easy to conflate with "consciousness" in philosophical discussions. It's not clear to me why that was the case since I gave a very specific definition of agency. 
*   Most of my students thought that AI models will never be conscious; it was difficult for them to articulate specific arguments about this, but their intuition seemed to be that there's something uniquely human about consciousness/sentience. 
*   The "AIs will take our jobs in the future" seems to be a very common concern both among students and academics. 
*   80% of a ~25 people classroom thought that philosophy is the right thing to major in if you're interested in how minds work. The question I asked them was: "should you major in philosophy or cognitive science if you want to study how minds work?"

### Governance/Strategy

*   Should we try to slow down AI progress? What does this mean in concrete steps? 
*   How should we go about capabilities externalities? 
*   How should concrete AI risk stories inform/affect AI governance and short-term/long-term future planning?