# [updated] how does gpt2′s training corpus capture internet discussion?  not well

\[**Updated** to correct my earlier claim that this doesn't affect GPT-3. Apparently it does?\]

I’m out sick today, but had enough energy to do some GPT-related fiddling around.

This time, I was curious what “internet discussions” tended to look like in the original training corpus.  I thought this might point to a more natural way to represent tumblr threads for [@nostalgebraist-autoresponder](undefined)​ than my special character trick.

So, I looked around in the large shard provided as part of [https://github.com/openai/gpt-2-output-dataset](https://github.com/openai/gpt-2-output-dataset).

[Colab notebook here](https://colab.research.google.com/drive/1Ka7teyOajTNP9LxE58zVg_NrGlks6DpX?usp=sharing), so you can interactively reproduce my findings or try similar things.

—–

The results were … revealing, but disappointing.  I did find a lot of discussion threads in the data (couldn’t find many chatlogs).  But

*   almost all of it is from phpBB-like forums (not _bad _per se, but weird)
*   it chooses a single post from each page and makes it “a text,” ignoring all the other posts, so no way for GPT2 to learn how users talk to each other :(
*   sometimes the post quotes another user… and in that case, you can’t see where the quote starts and the post begins
*   lots of hilarious formatting ugliness, like _“Originally Posted by UbiEpi Go to original post Originally Posted by”_
*   about 0.28% of the corpus (~22000 docs in full webtext) consists of these mangled forum posts
*   also, just as a chilling sidenote, about 0.30% of the corpus (~25200 docs in full webtext) is badly mangled pastebin dumps (all newlines removed, etc).no overlap between these and the mangled forum threads, so between them that’s ~0.58% of the corpus.
*   remember: the vast majority of the corpus is news and the like, so these percentages aren’t as small as they might sound

For example, from [this thread](http://www.pathofexile.com/forum/view-thread/991758/page/1) it picks the one post

![](https://64.media.tumblr.com/83ecb723cfaa92f163b5408889df911e/f23772776c08c401-1f/s540x810/a0323de0a351e49886b9aec2c3c3a1add62cead8.png)

and renders it as

> “ Pillowapnts  
>   
> tho the gem doesnt specifically say that you need to crit with something linked, i thought it was just crit in general ahhh. alright i get it thxtho the gem doesnt specifically say that you need to crit with something linked, i thought it was just crit in general  
>   
> That would be OP That would be OP Posted by Lordsidro  
>   
> on on Quote this Post

This is apparently standard behavior for the [newspaper](https://github.com/codelucas/newspaper) text cleaner they used, and I could reproduce it exactly.  (Its heuristics grab a single post when looking for the “part the content is in.”)  

\[_This paragraph was incorrect, see Update below_\] Does this affect GPT-3?  Probably not?  I don’t know how Common Crawl does text extraction, but at the very least, it’ll give you the whole page’s worth of text.

**Update:** Looked into this further, and I think GPT-3 suffers from this problem to some extent as well.

The Colab notebook has the details, but some stats here:

*   I used the small GPT-3 _samples_ dataset as a proxy for what GPT-3's corpus looks like (it's a skilled predictor of that corpus, after all)
*   The same filter that found malformed text in ~0.3% of the WebText shard also found it in 6/2008 (~0.3%) of GPT-3 samples.

*   Sample size here is small, but with 6 examples I at least feel confident concluding it's not a fluke.
*   They exhibit all the bad properties noted above. See for example [Sample 487](https://read-the-samples.netlify.app/sample_487) with "on on Quote this Post" etc.
*   For the official _GPT-2_ sample dataset (1.5B, no top-k), the filter matches 644/260000 samples (~0.25%, approximately the same).
*   In short, samples from both models produce "malformed forum" output at about the same rate as its actual frequency in the WebText distribution.

*   It's a bit of a mystery to me why GPT-3 and GPT-2 assign similar likelihood to such texts.

*   WebText was in the GPT-3 train corpus, but was only 22% of the distribution (table 2.2 in the paper).
*   Maybe common crawl mangles the same way? (I can't find good info on common crawl's text extraction process)
*   It's possible, though seems unlikely, that the "quality filtering" described in Appendix A made this worse by selecting WebText-like Common Crawl texts

*   On the other hand, the pastebin stuff is gone! (380/260000 GPT-2 samples, 0/2008 GPT-3 samples.) Your guess is as good as mine.
*   I'd be interested to see experiments with the API or AI Dungeon related to this behavior. _Can_ GPT-3 produce higher-quality forum-like output? Can it write coherent forum threads?