From: Christophe.Wolfhugel@grasp.insa-lyon.fr (Christophe Wolfhugel)
Newsgroups: news.lists
Subject: Usenet article size statistics -- August 92
Summary: statistical study of usenet articles
Keywords: article, statistics, size
Date: 4 Sep 92 05:45:21 GMT
Followup-To: news.admin.misc
Organization: INSA Informatique (Grasp), Lyon, France
NNTP-Posting-Host: grasp1.univ-lyon1.fr


-- New: at the end of the article a study of disk occupation.

Following statistics have been done using all 126405 articles having
transited on my node during the month of july 1992.

UUNET is also publishing a bi-weekly article size distribution
study to news.lists, but not yet with the per group details.

The sampled feed is far away from a full feed. The only binary
groups received are vmsnet. No pictures. No other binaries.

The sampled feed contains: news, comp (!sources,!binaries), misc.jobs,
bionet, vmsnet plus some other groups and some regional hierarchies
(about 300 groups total).


Number of articles received:           126405 articles
Smallest article received:                248 bytes
Largest article received:              380385 bytes
Average article size:                    2205 bytes
Standard deviation:                      4656 bytes

                          <-----August 1992 ---->    <- Jul 1992 ->
Article size       #        %       T#       T%         %       T%
-------------------------------------------------------------------
0-499            1341     1.06     1341     1.06      0.98     0.27
500-999         28777    22.77    30118    23.83     22.52    23.50
1000-1999       63124    49.94    93242    73.77     51.03    74.53
2000-4999       27595    21.83   120837    95.60     21.33    95.86
5000-9999        3187     2.52   124024    98.12      2.35    98.21
10000-24999      1397     1.10   125421    99.22      1.07    99.28
25000-49999       658     0.52   126079    99.74      0.50    99.78
50000+            326     0.26   126405   100.00      0.22   100.00


Following in an estimation of the "contents" of all sampled articles.
The sampled number of articles has 9 more, don't ask me
why... better than last time anyway :).

Number of articles sampled:        126414 articles

Field              Kb     %hdr    %nohdr
----------------------------------------
Kbytes:      272217.89   100.00   ---.--
Headers:      99491.38    36.55   ---.--
Body:        172726.51    63.45   100.00
Useful:      148598.67    54.59    86.03
Signatures:    9347.41     3.43     5.41
Quoting:      14780.43     5.43     8.56

Total top level domains:         149 (so many badly configured sites)

Sorted by size of articles posted...

     User address                               #msg       #kb    %sig  %quote
001. edu (1)                                   44672  101505.13    4.8    6.9
002. com (2)                                   33612   71128.36    5.2    9.5
003. ca (4)                                     6576   12299.69    5.3   10.3
004. uucp (3)                                   5911   10741.48    6.6   12.3
005. org (6)                                    4197    9792.06    5.3    9.7
006. de (8)                                     3916    9463.04    4.7    9.5
007. au (7)                                     4322    8615.91    6.0   12.4
008. uk (6)                                     3895    8118.71    5.5    9.1
009. gov (9)                                    2841    6122.21    8.1    7.9
010. us (-)                                     1890    3613.33    6.8    9.1
[...]

#msg   =   number of messages
#kb    =   size in kilobytes
%sig   =   percentage of (recognized) signatures
%quote =   percentage of (recognized) quoting.

The number in parenthesis, when present, indicates the rank of the
domain last month.

The complete set of data (one file per group and hierarchy or sub hierarchy
received here) is available either by anonymous ftp or listserv, and by
gopher (server: gopher.univ-lyon1.fr, port 70).

By anonymous ftp to grasp1.univ-lyon1.fr in /pub/usenet-stats, the file
'all' represents the global summary, 'news' for news.*, news.answers for
news.answers, ...

By listserv, send mail to listserv@grasp1.univ-lyon1.fr with commands in
the BODY of the message (not the subject!). Commands can be:

index usenet-stats              (approx. 30 Kb long message).
get usenet-stats group

Exemples:   get usenet-stats news
            get usenet-stats news.answers

Following algorithm has been used, it is far far away from being a good
one, but the results seem acceptable:

- headers are counted up to the first newline.

- signatures are counted from line '-- '. So exotic signatures are bypassed.

- default quote char is '>', if "In article" is encountered at a start
  of line, the quote char is set to one of <>:|%!.+~ on the first list
  it is encountered.

- top-level domains are determined from the part after the last dot in
  the address part, this is an interesting part as it allows to see
  some misconfigured sites, also, bang path addresses generate strange
  outputs. Some non standards lines have already been deleted by the
  statistics software.


Disk occupation statistics
--------------------------

These data indicates the disk usage of the articles on several
systems, depending on the block size.

This sampling has only be started recently, it only contains about
25000 articles, starting in September, the full traffic will be counted.

Sampled volume: 73427782 bytes (70.03 Mb)

Block size   #kb used   #kb lost    %
--------------------------------------
   512         80365      8658    10.8
  1024         88675     16968    19.1
  2048        103670     31963    30.8
  4096        158080     86373    54.6

Conlusion: get small blocking factors to save disk!

-- 
Christophe Wolfhugel    |    Email: Christophe.Wolfhugel@grasp.insa-lyon.fr
   "No keyboard, press F1 to continue"
Newsgroups: news.lists,news.groups