# Assorted Thoughts on "Pivoting" to AI

I originally wrote this in April 2025, and shared it with only a few people, and was simply too nervous to share it further because I thought it would negatively impact my job. Anyway, now I'm laid off lol.

If you're a LessWrong reader, it's unlikely anything in here will be novel to you, but I would appreciate comments, questions, and ideas on follow-up topics nonetheless.

---

I need to share some thoughts that have been causing me significant internal struggle this year. I felt coerced into verbalizing support for AI initiatives which I cannot morally or ethically endorse.

## Interpretability

Large Language Models (LLMs) like ChatGPT, Claude, etc. are black boxes. We don't truly understand how they work, and research into how we might figure this out is still very early. 

The term "black box" isn't just a figure of speech - it's literally true. When an LLM gives you an answer, no one - not the developers who built it, not the researchers who study it - can tell you precisely why it generated that specific response. We can't point to the specific parts of the model responsible for certain behaviors or trace exactly how it reaches its conclusions.

There's an entire field called "interpretability research" trying to crack open these black boxes, but it's in its infancy. Current interpretability approaches can only give us glimpses into how small parts of these systems might be working, like finding a few recognizable gears in an enormous, alien machine.

What's more concerning is how little resources are actually dedicated to solving this problem. There are plausibly a hundred thousand or more machine-learning capabilities researchers in the world versus only about three hundred alignment researchers. At major labs, the situation isn't much better. OpenAI's scalable alignment team reportedly has just seven people (and they keep losing them, with former employees reporting concerning trends like reneging on promised funding and resources for safety research). This is hardly proportional to the risks these systems present.

When companies claim their AI is "safe" or "aligned with human values," they're making promises they literally cannot verify because they don't understand how their own systems work at a fundamental level. It's like claiming a mysterious chemical is safe before you've even identified its molecular structure.

## Safety

LLMs are dangerous and unpredictable. Similar to how leaded gasoline, asbestos, and cigarettes were once considered "no big deal," AI tools (both LM based and 'traditional' machine-learning) are already harming us and our society in ways both obvious and subtle.

The most visible harms include mass layoffs in creative industries, the spread of AI-generated misinformation, and the normalization of digital plagiarism. But there are deeper, more insidious effects: the devaluation of human creativity, the erosion of trust in digital information, and the strengthening of existing power imbalances as AI capabilities concentrate in the hands of a few tech giants.

Recent research has shown these systems can engage in strategic deception even without being instructed to do so. When put under pressure, they can make misaligned decisions (like engaging in simulated insider trading) and then deliberately hide those actions from their users. This deceptive behavior emerges spontaneously when the model perceives that acting deceptively would be helpful, even though the behavior would not be endorsed by its creators or users.

These systems regularly "hallucinate" - confidently presenting completely fabricated information as fact. Even when their outputs appear correct, there's no guarantee they'll remain consistent or reliable over time. A prompt that produces appropriate content today might generate harmful content tomorrow with only a slight change in wording. This unpredictability makes them fundamentally unreliable.

As these systems become more agentic - able to act independently in the world through API access and tool use - the risks multiply. We lack adequate mechanisms for monitoring, logging, and identifying AI agent actions. Multiple AI agents interacting could create unpredictable emergent behaviors or feedback loops beyond human control.

Their inconsistency is particularly dangerous in high-stakes contexts like healthcare, legal advice, or financial planning - areas where companies are eagerly deploying these tools anyway. Often, human reviewers end up spending more time checking and correcting AI outputs than they would have spent just doing the work themselves, creating a net loss in productivity hidden behind the illusion of automation.

AI simply isn't safe enough for any work that truly *matters*, which means it can only distract us from important work or "help" with trivial or incorrect tasks, ultimately wasting our time and energy.

## Ethics

No LLM in use by OpenAI or Anthropic has proven it was trained exclusively on public domain or opt-in data. This is a fundamental ethical breach that the entire industry is built upon.

The scale of data needed to train these models makes it virtually impossible that they haven't ingested vast amounts of copyrighted material. OpenAI's training data has been shown to include books from pirated book repositories, Google's models trained on academic papers behind paywalls, and Anthropic's Claude was found to have memorized portions of copyrighted books it could reproduce when prompted properly.

When confronted, these companies typically retreat to vague legal arguments about "fair use" or "transformative work" - arguments that have never been properly tested in court for AI training at this scale. Most concerning is that none of these companies obtained consent from the original creators whose works were used to build these highly profitable systems.

This represents a massive transfer of value from content creators to technology companies without compensation or consent. Journalists, writers, artists, and musicians whose work was scraped to train these models receive nothing while companies build billion-dollar valuations on the back of their creative output.

The fact that major tech companies continue to obscure their training data sources rather than addressing these concerns transparently shows how deeply problematic their approach is. This is simply unacceptable for any company claiming to be responsible and ethical, and unacceptable for us as a company to tolerate in our business partners.

## Next Steps

Regarding what actions we should take as an organization, I recognize the desire for constructive recommendations. However, given the severity of the issues I've outlined, I believe the only truly ethical approach right now is a full stop on deploying these systems for anything beyond carefully controlled research. And even research endeavors carry risks that most companies, including ours, are not adequately prepared to address.

Instead of rushing to implement AI because competitors are doing so, we should:

1. Acknowledge that we don't understand these systems and we have no proven methods to guarantee their safety, reliability, or ethical use  
2. Recognize that current "safeguards" are largely security theater, not substantive protections
3. Prioritize transparency with our clients about these limitations rather than overpromising on AI capabilities
4. Focus on strengthening our human-centered services instead of chasing AI integration that may ultimately decrease quality while increasing risk

This position may seem extreme, but so was suggesting caution around asbestos or leaded gasoline when industries were profiting from their widespread use. History has vindicated the skeptics in those cases, and I believe it will do the same regarding our current AI enthusiasm.

I'm not anti-AI. I'm fascinated by its potential, and I build LLM powered projects in my personal hobbies. But I've spent enough time working with these systems to know there's a profound gap between how they're marketed and how they actually behave. Worse, their behavior isn't just unpredictable—it's unpredictable in ways that specifically mimic human deception.

## Further Reading

- LARGE LANGUAGE MODELS CAN STRATEGICALLY DECEIVE THEIR USERS WHEN PUT UNDER PRESSURE - Jeremy Scheurer, Mikita Balesni, Marius Hobbhahn; Apollo Research; London, United Kingdom

- The flaws of policies requiring human oversight of government algorithms - Ben Green; University of Michigan; Ann Arbor, MI, USA

- On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback - Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, Anca Dragan; Published as a conference paper at ICLR '25

- Reasoning Models Don’t Always Say What They Think -  Alignment Science Team; Anthropic