From: Christophe.Wolfhugel@grasp.insa-lyon.fr (Christophe Wolfhugel)
Newsgroups: news.lists
Subject: Usenet article size statistics -- December 1992
Followup-To: news.admin.misc
Date: 6 Jan 1993 13:44:28 +0100
Organization: INSA Informatique (Grasp), Lyon, France
NNTP-Posting-Host: grasp1.univ-lyon1.fr
Summary: statistical study of usenet articles
Keywords: article, statistics, size


Following statistics have been done using all articles having
transited on my node during the month of December 1992.

UUNET is also publishing a bi-weekly article size distribution
study to news.lists. Matthias Urlichs publishs a detailled list
on a per group basis.  None of those postings do an analysis
based on sender's addresses, this is what is included below.

The sampled feed is far away from a full feed. The only binary
groups received are vmsnet. No pictures. No other binaries.

The sampled feed contains: bit, news, comp (!sources,!binaries), misc.jobs,
bionet, vmsnet plus some other groups and some regional hierarchies
(about 700 groups total).


Number of articles received:           143861 articles
Smallest article received:                246 bytes
Largest article received:              297640 bytes
Average article size:                    2167 bytes
Standard deviation:                      4555 bytes

Article size       #        %       T#       T%
------------------------------------------------
0-499            1623     1.31     1623     1.31
500-999         34748    23.97    36371    25.28
1000-1999       70539    49.03   106910    74.31
2000-4999       30489    21.20   137399    95.51
5000-9999        3911     2.72   141310    98.23
10000-24999      1518     1.05   142828    99.28
25000-49999       696     0.49   143524    99.77
50000+            337     0.03   143861   100.00

Following in an estimation of the "contents" of some sampled articles.
The difference is explained by articles having been cancelled, those
cross-posted to groups where the first listed group is not received here.

Total articles:     139375

Field              Kb     %hdr    %nohdr
----------------------------------------
Kbytes:     286002.47   100.00    ---.--
Headers:    120801.78    42.24    ---.--
Body:       165200.70    57.76    100.00
Useful:     141318.09    49.41     85.54
Signatures:   7873.03     2.75      4.77
Quoting:     16009.57     5.60      9.69

       User address                             #msg        #kb    %sig  %quote
00000. .edu                                    48685  101303.08    2.50    5.24
00001. .com                                    31731   62889.73    3.51    6.81
00002. .net                                     5907   20139.57    0.34    0.60
00003. .ca                                      7289   13560.05    3.83    6.21
00004. .org                                     4516    9643.55    2.32    5.24
00005. .uk                                      4975    8802.13    3.69    6.40
00006. .de                                      4223    8458.87    3.60    7.07
00007. .au                                      3502    7555.75    3.74    7.84
00008. .bitnet                                  4228    7001.51    0.06    4.23
00009. .uucp                                    3446    6000.70    3.98    7.38
00010. .gov                                     2693    5799.10    2.54    4.73
00011. .se                                      1661    3628.61    3.51    5.80
00012. .us                                      1805    3353.17    3.21    7.12
00013. .nl                                      1664    3336.01    3.21   10.04
00014. .mil                                     1405    3205.18    3.41    7.75
00015. UNPARSEABLE                              1944    3176.84    1.53    4.17
00016. .fi                                      1459    3028.46    1.71    6.83
00017. .nz                                      1309    2482.88    3.64    7.58
00018. .fr                                       833    1787.21    3.44    4.03
00019. .no                                       852    1610.74    2.72    5.47
00020. .ch                                       794    1460.89    3.50    3.55
00021. .dk                                       467     926.16    2.14   10.24
00022. .jp                                       331     617.60    1.62    3.57
00023. .sg                                       417     564.32    0.89    1.19
00024. .it                                       232     509.97    9.62    3.51
00025. .ie                                       312     508.39    2.67    5.95
00026. .at                                       285     489.18    2.25    2.95
00027. .be                                       239     484.01    8.22    1.56
00028. .tw                                       276     403.43    0.30    0.26
00029. .su                                       164     363.06    2.16    1.80
00030. .il                                       237     329.53    0.82    4.21
[...]

#msg   =   number of messages
#kb    =   size in kilobytes
%sig   =   percentage of (recognized) signatures
%quote =   percentage of (recognized) quoting.

The number in parenthesis, when present, indicates the rank of the
domain last month.

The complete set of data (one file per group and hierarchy or sub hierarchy
received here) is available either by anonymous ftp or listserv, and by
gopher (server: gopher.univ-lyon1.fr, port 70).

By anonymous ftp to grasp1.univ-lyon1.fr in /pub/usenet-stats, the file
'all' represents the global summary, 'news' for news.*, news.answers for
news.answers, ...

By listserv, send mail to listserv@grasp1.univ-lyon1.fr with commands in
the BODY of the message (not the subject!). Commands can be:

index usenet-stats              (approx. 50 Kb long message).
get usenet-stats group

Exemples:   get usenet-stats news
            get usenet-stats news.answers

Following algorithm has been used, it is far far away from being a good
one, but the results seem acceptable:

- headers are counted up to the first newline.

- signatures are counted from line '-- '. So exotic signatures are bypassed.

- default quote char is '>', if "In article" is encountered at a start
  of line, the quote char is set to one of <>:|%!.+~ on the first list
  it is encountered.

- top-level domains are determined from the part after the last dot in
  the address part, this is an interesting part as it allows to see
  some misconfigured sites, also, bang path addresses generate strange
  outputs. Some non standards lines have already been deleted by the
  statistics software.

Note that exceptionnally this month, cross-posted articles have only
been charged to the first group listes in the Newsgroups: header.


Disk occupation statistics
--------------------------

These data indicates the disk usage of the articles on several
systems, depending on the block size.

All articles received in October have been included in this sample.

Sampled volume: 311851564 bytes (297.4 Mb)

Block size   #kb used   #kb lost    %       System name
-------------------------------------------------------
   512        340665     36122    10.6      [ Put here
  1024        374347     69804    18.6        your
  2048        438262    133719    30.5      favorite
  4096        668204    363661    54.4      system name ]

Conlusion: get small blocking factors to save disk!

-- 
Christophe Wolfhugel    |    Email: Christophe.Wolfhugel@grasp.insa-lyon.fr
Xref: sparky news.lists:559 news.announce.newusers:214 news.answers:4923