External storage/Text storage data

== Raw Data ==
Text row types as of 2010-02-18. All databases.
Count Type
------------------------------------------------
9 0,external/simple pointer
435 0/[none]
1482941 [none]/[none]
74069 external,object/simple pointer
56103275 external,utf-8/CGZ pointer
329027392 external,utf-8/DHB pointer
472766 external,utf8/CGZ pointer
7409721 external,utf8/DHB pointer
2890300 external/CGZ pointer
12218 external/simple pointer
4113780 gzip,external/simple pointer
968957 gzip/[none]
178234 object,external/simple pointer
387 object,utf-8/ConcatenatedGzipHistoryBlob
1413 object,utf-8/HistoryBlobStub
216694 object/concatenatedgziphistoryblob
5842435 object/historyblobcurstub
1121994 object/historyblobstub
1 utf-8,external/simple pointer
464549188 utf-8,gzip,external/simple pointer
17076928 utf-8,gzip/[none]
1269 utf-8/[none]
Text row types as of 2011-08-31. All wikis:
(see RT:1300 for details on how these numbers were generated.)
 Count Type
------------------------------------------
 9 0,external/simple pointer
 437 0/[none] 
 1473028 [none]/[none] 
 64751783 external,utf-8/CGZ pointer
 363080206 external,utf-8/DHB pointer
 484000 external,utf8/CGZ pointer
 7579959 external,utf8/DHB pointer
 1180404 external/CGZ pointer
 12218 external/simple pointer
 9905103 gzip,external/simple pointer
 968337 gzip/[none] 
 181328 object,external/simple pointer
 387 object,utf-8/ConcatenatedGzipHistoryBlob 
 1413 object,utf-8/HistoryBlobStub 
 219570 object/concatenatedgziphistoryblob 
 5866400 object/historyblobcurstub 
 1046202 object/historyblobstub 
 1 utf-8,external/simple pointer
 797612169 utf-8,gzip,external/simple pointer
 17173393 utf-8,gzip/[none] 
 1269 utf-8/[none] 
Change from 2010-02-18 to 2011-08-31:
 2010-02-18 2011-08-31 Diff Type
---------------------------------------------------------------------
 9 9 0 0,external/simple pointer
 435 437 2 0/[none] 
 1482941 1473028 -9913 [none]/[none] 
 74069 n/a n/a external,object/simple pointer
 56103275 64751783 8648508 external,utf-8/CGZ pointer
 329027392 363080206 34052814 external,utf-8/DHB pointer
 472766 484000 11234 external,utf8/CGZ pointer
 7409721 7579959 170238 external,utf8/DHB pointer
 2890300 1180404 -1709896 external/CGZ pointer
 12218 12218 0 external/simple pointer
 4113780 9905103 5791323 gzip,external/simple pointer
 968957 968337 -620 gzip/[none] 
 178234 181328 3094 object,external/simple pointer
 n/a 387 n/a object,utf-8/ConcatenatedGzipHistoryBlob 
 n/a 1413 n/a object,utf-8/HistoryBlobStub 
 1800 n/a n/a object,utf-8/[none] 
 216694 219570 2876 object/concatenatedgziphistoryblob 
 5842435 5866400 23965 object/historyblobcurstub 
 1121994 1046202 -75792 object/historyblobstub 
 1 1 0 utf-8,external/simple pointer
 464549188 797612169 333062981 utf-8,gzip,external/simple pointer
 17076928 17173393 96465 utf-8,gzip/[none] 
 1269 1269 0 utf-8/[none] 
== Analysis ==
=== On the changes from 2010-2011 ===
The rise in "object/historyblobcurstub" doesn't really make sense. 
The rise in "gzip,external/simple pointer" is concerning.
== Description of fields and values ==
; [none]/[none] : Uncompressed text, legacy encoding
; 0/[none] : Uncompressed text, wrong flags due to short-lived bug, never cleaned up
; 0,external/simple pointer : As above plus MTE
; gzip/[none] : Compressed text with legacy encoding. Possibly created with Brion's original CO.
; gzip,external/simple pointer : As above plus MTE
; utf-8/[none] : Uncompressed MW 1.5+
; utf-8,external/simple pointer : As above plus MTE
; utf-8,gzip/[none] : Compressed MW 1.5+, probably generated directly by MW
; utf-8,gzip,external/simple pointer : Either as above plus MTE, or directly generated by MW (predominant non-recompressed type)
; object,utf-8/ConcatenatedGzipHistoryBlob : Presumably created by a brief enwiki-only run of CO, in MW 1.5+.
; object,utf-8/HistoryBlobStub : Stubs for the above CO run
; object/concatenatedgziphistoryblob : Object created by CO, MW<1.5
; object,external/simple pointer : As above plus MTE
; object/historyblobcurstub : Created by the 1.5 upgrade script, a reference to the cur table.
; object/historyblobstub : Pointer to a CGZ object, created by CO, MW<1.5
; external,object/simple pointer : Possibly JOMTE 
; external/simple pointer : JOMTE?
; external,utf-8/CGZ pointer : Late CO, RS or RCT
; external,utf-8/DHB pointer : RCT
; external,utf8/CGZ pointer : RCT with buggy encoding name, <r45205
; external,utf8/DHB pointer : RCT <r45205
; external/CGZ pointer : RS. Perhaps CO in MW<1.5 also created these.
=== Legend ===
; CO : compressOld.php.
; MTE : moveToExternal.php. 
; MW : MediaWiki
; JOMTE: JeLuF's original move to external. I think there was an SQL script or something that he used to move some text when external storage was set up initially, I can't find it now.
; RCT : recompressTracked.php. The latest and greatest recompression script.
; RS : resolveStubs.php
== How these stats were generated ==
storageTypeStatsDiff.py and storageTypeStatsSum.py exist in svn
To collect the stats, gather info for every wiki db (this step takes about 24 hours):
 ben@hume:~$ cd /home/w/bin/
 ben@hume:bin$ ./foreachwiki maintenance/storage/storageTypeStats.php > /tmp/storageTypeStats.log
 ben@hume:bin$ scp /tmp/storageTypeStats.log fenari:
To sum the stats for each wiki, this output is sent through storageTypeStatsSum.py:
 ben@fenari:~$ cd svn/extensions/WikimediaMaintenance/storage/
 ben@hume:storage$ ./storageTypeStatsSum.py ~/storageTypeStats.log > current-YYYY-MM-DD
To calculate the differences, grab the previous stats from this page, store them in a date-named file and compare them:
 ben@hume:storage$ cat <<EOOLDSTATS > <old-date>
 <paste in content from this wiki page>
 EOOLDSTATS
 ben@hume:storage$ ./storageTypeStatsDiff.py <old-date> <current-date> > /tmp/storageDiffs.log
 ben@hume:storage$ rm <old-date> <current-date>
paste the new values and the diff into this wiki page
== Bugs ==
* Bug 950: botched conversion from latin1 to UTF-8 on es.wiktionary.org. See the historical worksheet compression corruption.
* bug 22624 compressOld.inc with CGZ may have been run as early as October 2004. It wasn't until December 2004 that r6640 was committed, which prevented CGZ blobs from being moved to the archive table. The English Wikipedia archive table now has 892 CGZ blobs, 1541 HistoryBlobStub objects, and 510 "external,object" rows. These all need to be fixed urgently, since RCT will destroy them.