Summarize this and list important information as bullet points: 

AI
OpenAI’s new tool attempts to explain language models’ behaviors
Kyle Wiggers
@kyle_l_wiggers / 6:00 PM GMT+1•May 9, 2023

OpenAI's logo
Image Credits: OpenAI
It’s often said that large language models (LLMs) along the lines of OpenAI’s ChatGPT are a black box, and certainly, there’s some truth to that. Even for data scientists, it’s difficult to know why, always, a model responds in the way it does, like inventing facts out of whole cloth.

In an effort to peel back the layers of LLMs, OpenAI is developing a tool to automatically identify which parts of an LLM are responsible for which of its behaviors. The engineers behind it stress that it’s in the early stages, but the code to run it is available in open source on GitHub as of this morning.

“We’re trying to [develop ways to] anticipate what the problems with an AI system will be,” William Saunders, the interpretability team manager at OpenAI, told TechCrunch in a phone interview. “We want to really be able to know that we can trust what the model is doing and the answer that it produces.”


To that end, OpenAI’s tool uses a language model (ironically) to figure out the functions of the components of other, architecturally simpler LLMs — specifically OpenAI’s own GPT-2.

OpenAI explainability
OpenAI’s tool attempts to simulate the behaviors of neurons in an LLM. Image Credits: OpenAI


How? First, a quick explainer on LLMs for background. Like the brain, they’re made up of “neurons,” which observe some specific pattern in text to influence what the overall model “says” next. For example, given a prompt about superheros (e.g. “Which superheros have the most useful superpowers?”), a “Marvel superhero neuron” might boost the probability the model names specific superheroes from Marvel movies.

OpenAI’s tool exploits this setup to break models down into their individual pieces. First, the tool runs text sequences through the model being evaluated and waits for cases where a particular neuron “activates” frequently. Next, it “shows” GPT-4, OpenAI’s latest text-generating AI model, these highly active neurons and has GPT-4 generate an explanation. To determine how accurate the explanation is, the tool provides GPT-4 with text sequences and has it predict, or simulate, how the neuron would behave. In then compares the behavior of the simulated neuron with the behavior of the actual neuron.

“Using this methodology, we can basically, for every single neuron, come up with some kind of preliminary natural language explanation for what it’s doing and also have a score for how how well that explanation matches the actual behavior,” Jeff Wu, who leads the scalable alignment team at OpenAI, said. “We’re using GPT-4 as part of the process to produce explanations of what a neuron is looking for and then score how well those explanations match the reality of what it’s doing.”

The researchers were able to generate explanations for all 307,200 neurons in GPT-2, which they compiled in a dataset that’s been released alongside the tool code.

Tools like this could one day be used to improve an LLM’s performance, the researchers say — for example to cut down on bias or toxicity. But they acknowledge that it has a long way to go before it’s genuinely useful. The tool was confident in its explanations for about 1,000 of those neurons, a small fraction of the total.

A cynical person might argue, too, that the tool is essentially an advertisement for GPT-4, given that it requires GPT-4 to work. Other LLM interpretability tools are less dependent on commercial APIs, like DeepMind’s Tracr, a compiler that translates programs into neural network models.

Wu said that isn’t the case — the fact the tool uses GPT-4 is merely “incidental” — and, on the contrary, shows GPT-4’s weaknesses in this area. He also said it wasn’t created with commercial applications in mind and, in theory, could be adapted to use LLMs besides GPT-4.

OpenAI explainability
The tool identifies neurons activating across layers in the LLM. Image Credits: OpenAI


“Most of the explanations score quite poorly or don’t explain that much of the behavior of the actual neuron,” Wu said. “A lot of the neurons, for example, are active in a way where it’s very hard to tell what’s going on — like they activate on five or six different things, but there’s no discernible pattern. Sometimes there is a discernible pattern, but GPT-4 is unable to find it.”

That’s to say nothing of more complex, newer and larger models, or models that can browse the web for information. But on that second point, Wu believes that web browsing wouldn’t change the tool’s underlying mechanisms much. It could simply be tweaked, he says, to figure out why neurons decide to make certain search engine queries or access particular websites.

“We hope that this will open up a promising avenue to address interpretability in an automated way that others can build on and contribute to,” Wu said. “The hope is that we really actually have good explanations of not just what neurons are responding to but overall, the behavior of these models — what kinds of circuits they’re computing and how certain neurons affect other neurons.”

More TechCrunch


Google's 'Sidekick' offers contextual prompts to get better results from AI

Theranos founder Elizabeth Holmes isn't headed to jail tomorrow after all

American psychology group issues recommendations for kids' social media use

OpenAI’s ChatGPT is shaking up the edtech markets

Please login to comment
Login / Create Account

TechCrunch Disrupt
Sept 19-21 San Francisco, CA
Register Now
Sign up for Newsletters
See all newsletters(opens in a new window)


Daily

Week in Review

Startups Weekly

Event Updates

Advertising Updates

TechCrunch+ Announcements

TechCrunch+ Events

TechCrunch+ Roundup

Email
Subscribe
(opens in a new window)(opens in a new window)(opens in a new window)(opens in a new window)(opens in a new window)
AI
OpenAI’s new tool attempts to explain language models’ behaviors
Kyle Wiggers
6:00 PM GMT+1•May 9, 2023
OpenAI's logo
AI
EU lawmakers back transparency and safety rules for generative AI
Natasha Lomas
11:18 AM GMT+1•May 11, 2023
Artificial Intelligence for Deep Learning Technology over Top view scene of Motion blurred Crowd unrecognizable pedestrians walking internal subway intersection in rush hour before working hour, central Hong Kong.
Gadgets
Canon sidles up to vloggers with PowerShot V10
Haje Jan Kamps
10:50 AM GMT+1•May 11, 2023

Apps
Twitter launches encrypted DMs for verified users with security drawbacks
Ivan Mehta
10:34 AM GMT+1•May 11, 2023
distorted, painterly twitter logo with a background of gears
Gadgets
Fairphone gets its audio groove on with repairable over-ear BT headphones
Natasha Lomas
10:00 AM GMT+1•May 11, 2023
Fairphone Fairbuds XL modular headphones in black - exploded view
Startups
Microsoft makes strategic investment into Builder.ai, integrates its services into Teams
Mike Butcher
9:00 AM GMT+1•May 11, 2023

Apps
Google’s new Labs page lets you sign up for its AI experiments
Ivan Mehta
8:49 AM GMT+1•May 11, 2023
Google Labs
Venture
SoftBank Vision Fund yearly loss widens to $32B on startups valuation cut
Manish Singh
8:06 AM GMT+1•May 11, 2023
SoftBank Vision Fund loses $32 billion in a year as startups confront valuation cut
AI
Everseen raises over $70M for AI tech to spot potential retail theft
Kyle Wiggers
8:00 AM GMT+1•May 11, 2023

Transportation
Wingcopter, Germany’s drone delivery startup, raises another $44M from the EIB
Ingrid Lunden
7:20 AM GMT+1•May 11, 2023

Media & Entertainment
Disney’s Hotstar loses 8.4 million subscribers as Ambani’s JioCinema ascends
Manish Singh
6:16 AM GMT+1•May 11, 2023
Ambani bats for cricket glory as Disney scales back
TechCrunch Disrupt 2023
APAC startups: Apply to Startup Battlefield 200
Lauren Simonds
1:00 AM GMT+1•May 11, 2023

Startups
Daily Crunch: OpenAI, Anthropic and Stability AI receive half of Sound Ventures’ $240M AI fund
Christine Hall, Haje Jan Kamps
11:05 PM GMT+1•May 10, 2023
CVC, corporate venture capital,
Media & Entertainment
Disney+ and Hulu content to combine into one streaming app
Lauren Forristal
10:10 PM GMT+1•May 10, 2023
Disney+ app on phone
AI
Google I/O 2023 is a wrap — here’s a list of everything announced
Christine Hall
9:39 PM GMT+1•May 10, 2023
Google CEO Sundar Pichai presents at Google I/O 2023
Apps
Google ‘Perspectives’ integrates Reddit, YouTube, TikTok and more in search results
Sarah Perez
9:31 PM GMT+1•May 10, 2023
google search app ios
Startups
The Mint, started by Better Tomorrow Ventures, wants to be the accelerator fintech needs
Natasha Mascarenhas
9:30 PM GMT+1•May 10, 2023
deal pace picking up
Media & Entertainment
Disney+ loses subscribers for second quarter in a row, drops 4M subs
Lauren Forristal
9:18 PM GMT+1•May 10, 2023

Transportation
Google is bringing YouTube to more cars, starting with Polestar
Rebecca Bellan
9:01 PM GMT+1•May 10, 2023
YouTube on display screen of Polestar vehicle
Transportation
Google adding Cisco, Microsoft and Zoom (audio) conferencing to Android Auto
Rebecca Bellan
9:00 PM GMT+1•May 10, 2023
Zoom android auto google
Transportation
All the ways Google is driving deeper into the automotive world
Rebecca Bellan, Kirsten Korosec
9:00 PM GMT+1•May 10, 2023
Waze Google built in car
About
TechCrunch
Staff
Contact Us
Advertise
Crunchboard Jobs
Site Map
Legal
Terms of Service
Privacy Policy
TechCrunch+ Terms
Privacy Dashboard
Code of Conduct
About Our Ads
Trending Tech Topics
Tech Layoffs
ChatGPT
Facebook(opens in a new window)Twitter(opens in a new window)YouTube(opens in a new window)Instagram(opens in a new window)LinkedIn(opens in a new window)Mastodon(opens in a new window)
© 2023 Yahoo.
All rights reserved.
Powered by WordPress VIP(opens in a new window).