Talk:Zipf's law

message: The article needs to start with a very simple leading statement, so that anyone stumbling onto it can know what it is about. The discussion can then move onto more technical stuff.
* Provided one. --Jorge Stolfi (talk) 01:24, 9 May 2023 (UTC)
thread_title: Needs simple leading statement

message: Is it true that the word "the" does indeed occur about twice as often as the next common English word? The rest of the article seems to allow for some proportionality constants. AxelBoldt
* It's not true, so I replaced it with a statement about Shakespeare's plays. AxelBoldt
** It is approximately true, see the example of the Brown corpus. The normalization constant applies to both words, so the ratio is 1/1 to 1/2, that is 2:1.  --Jorge Stolfi (talk) 01:28, 9 May 2023 (UTC)
thread_title: True that "the" is twice as common as the next one?

message: The main article claimed that
* the frequency distribution of words generated by random typing follows Zipf's law.
I doubt that very much. For one thing, if you type randomly, all words of length one will be equally likely, all words of length 2 will be equally likely and so on. Or am I missing something? Maybe we should perform a little perl experiment. AxelBoldt
::I tried this Python code, and plotted the results in a log-log plot.  The early ranks are a bit stepped, but the overall pattern fits Zipf's law rather well. The Anome
::
import random
import string
import math
N=10000
M=100
words = {}
for j in range(M):
  str = []
  for i in range(N):
    str.append(random.choice('aaabccdefg     '))
  str = string.join(str, )
  str = string.split(str)
  for word in str:
    if words.has_key(word):
        words[word] += 1
    else:
        words[word] = 1
  print 'did string pass', j
  
vals = words.values()
vals.sort()
vals.reverse()
file = open('zipf_ranks.txt', 'w')
# Let's just have the first few ranks
useranks = min(1000, len(vals))
for i in range(useranks):
    rank = i+1
    file.write("%d %d\n" % (rank, vals[rank-1]))
----
:::Could you repeat the experiment with all letters and the space getting the same probability? That's at least what I thought off when I heard "random typing". AxelBoldt
::::When I was told of the "random typing" experiment, the "typing" part was more important than "random".  If you sit at a keyboard and randomly type, you have a much higher chance of hitting certain keys than others because your fingers tend to like certain positions.  Also, the human brain is a pattern-matching machine, so it works in patters.  If you look at what you type, you might notice you tend to type the same sequences of letters over and over and over.
::::For a brief historical note, this theory was used in cracking the one-time-pad cyphers.  Humans typed the pads, so it was possible to guess at the probability of future cyphers by knowing past ones.  It is like playing the lottery knowing that a 6 has a 80% chance of being the first number while a 2 has a 5% chance.  If you knew the percentage chance for each number and each position, you have a much greater chance of winning over time. Kainaw 13:59, 24 Sep 2004 (UTC)
replies: text: this is a pretty minor example of Zipf's law. A much more important one is that the frequency of occurrence of words in natural language conform to Zipf's law. This is in the original 1949 reference. Callivert (talk) 11:29, 24 March 2008 (UTC)
thread_title:  Random typing 

message: Moved from main article:
:We need an explanation here: why do these distributions follow Zipf's law?
::No we don't.  Zipf's law is empirical, not theoretical.  We don't know why it works.  But even without a theory, even the simplest experiments that try to model a society of independent actors consistently turn it up!--BJT
Well, empirical facts have to be explained too. It's  not enough to simply state that the moon always shows us the same side; you have to give the reason if you try to understand the world. It's the same here. If Zipfian distributions show up in a variety of situations, then there must be some underlying principle which generates them. I doubt very much that "we don't know" that principle. AxelBoldt
: I agree - every theory begins with empirical evidence. The theory models an explanation to fit those facts. I'm sure someone has tried to come up with an explanation? 70.93.249.46
----
:We need an explanation here: why do these distributions follow Zipf's law?
An excellent question.
However, I doubt there is a single cause that can explain every occurance of Zipf's law.
(For some distributions, such as wealth distribution, the cause of the distribution is controversial).
Well, empirical facts have to be explained too. It's  not enough to simply state that the moon always shows us the same side; you have to give the reason if you try to understand the world.
Good point.
However, sometimes we don't yet know the cause of some empirical facts -- we can't yet give a good explanation.
In those cases, I would prefer the Wikipedia article to bluntly tell me "we don't know yet" rather than try to dance around that fact.
----
While it is true that Zipf's law is empirical, I agree with AxelBoldt that it is useful to have an interpretation of it.  The most obvious place to look is the book that Zipf himself wrote in which he linked his observation to the Principle of Least-Effort, kind of an application of Conservation of Energy to human behavior.  
----
I've just measured Polish Wikipedia page access distribution using Apache logs (so they had to be heavily Perlscripted) for about 2 weeks of late July, only for main namespace articles, and excluding Main Page. In most part it seems to be following Zipf's law with b about 0.5, except at both ends, where it behaves a bit weird (what was to be expected). Now why did I get constant so grossly different from stated constant for English Wikipedia ?
Some possibilities:
* Polish and English Wikipedias really have different Zipf's factors
* It was due to my perlscripting
* Measurment given for English Wikipedia here is wrong for some reason, like measuring only top 100, and not all articles.
Taw 01:38, 4 Aug 2003 (UTC)
thread_title:  Why ? 

message: the examples section reports values of s < 1 as resulting from analysis of wikipedia page view data.
the earlier discussion correctly notes that such values do not yield a valid probability distribution.
what gives? perhaps (s - 1) is being reported?
:Or it could be that that value of s is right for a moderately large (hundreds?) finite number of pages.  That seems to happen with some usenet posting statistics. Michael Hardy 22:07, 28 Jun 2004 (UTC)
thread_title:  reported constants in Examples section 

message: I plotted Shakespeare's word frequency lists and top 5000 words in Wikipedia:  Where did the old value of s~0.5 come from? -- Nichtich 00:27, 23 Jun 2005 (UTC)
thread_title:  ''s'' = 1? 

message: Section 3 ("Examples of collections approximately obeying Zipf's law") has a bunch of examples with no further explanation or reference (Shakespeare excluded). That's highly undesirable, so let's get some sources. By the way, I find the final point (notes in a musical performance) very questionable. I imagine it would depend very much on the type of music. EldKatt (Talk) 13:20, 19 July 2005 (UTC)
replies: text: It might not be a bad idea to give examples from Zipf's book.  Also see the article by Richard Perline in the February 2005 issue of Statistical Science. Michael Hardy 22:28, 19 July 2005 (UTC)
thread_title:  Sources needed for examples 

message: Rd232 added the "technical" template to the article. I've moved it here per template. Paul August ☎ 03:49, 27 November 2005 (UTC)
replies: text: There have been numerous edits to the article since the template was first added almost a year ago.  Also, there has been no discussion of what about the article is too technical so I've removed the template.  Feel free to put it back, but if you do, please leave some comments as to what you find is too technical and some suggestions as to how to improve the article.  Lunch 02:25, 21 November 2006 (UTC)
thread_title:  Too technical? 

message: Yes, apparently, with an exponent of 0.5. See Wikipedia:Does Wikipedia traffic obey Zipf's law? for more. -- The Anome 22:45, 20 September 2006 (UTC)
replies: text: Why 0.5? And what else has the exponent? I've got 0.5 in some genomic dta, don't know why its 0.5 and not 1.0 .... 67.198.37.16 (talk) 18:17, 24 February 2020 (UTC)
thread_title:  Does Wikipedia traffic obey Zipf's law? 

message: Just a plot of English Wikipedia word frequencies: http://oc-co.org/?p=79
: Is this plot availiable to Wikipedia - i.e. Free content? It would look good in the article.
:: Yes, it is released under LGPL by me, the author :)  --  Victor Grishchenko
::: I downloaded it, tagged it as LGPL with you as author, and put it into the article - please check it out and make corrections if needed. This is an excellent demonstration of Zipf's law (and its limitations). Thanks! PAR 15:00, 29 November 2006 (UTC)
replies: text: It's a general principle that words drop off more quickly than Zipf's law predicts. If we looked at word n-grams I think we would see something closer. Dcoetzee 21:27, 31 October 2008 (UTC)

text: Index wiki database: design and experiments. Krizhanovsky A. A. In: FLINS'08, Corpus Linguistics'08, AIS/CAD'08, 2008. -- AKA MBG (talk) 17:52, 3 November 2008 (UTC)
thread_title:  Wikipedia's Zipf law 

message: I'm not sure if k=0 should be included in the support... pmf is not well defined there, as 1 / (k^s) = 1 / 0 and it -> +inf. 
Krzkrz 08:56, 3 May 2007 (UTC)
thread_title:  k = 0 in support? 

message: I was surprised that there wasn't even a brief note at the beginning of the article on who Zipf is (was?).  I think it's sort of nice to see that before you get into the technical stuff.
Jdrice8 05:38, 14 October 2007 (UTC)
replies: text: That was the brief second paragraph of the article.  Now I've made it into a parenthesis in the first sentence, set off by commas. Michael Hardy 01:58, 15 October 2007 (UTC)
thread_title:  biographical information 

message: I'm sure I've come across in a no. of places the term Zipf's law used to refer to the inverse relation between word length & frequency. This may well be a misuse, but isn't it common enough to be mentioned with a link to whatever the correct term is? Peter jackson (talk) 14:43, 15 September 2008 (UTC)
thread_title:  Word length 

message: The lead must be rephrased: defining Zipf's law in terms of a Zipfian distribution is no help to the reader who must actually learn. Srnec (talk) 15:11, 31 October 2008 (UTC)
replies: text: I was just coming by to say the same thing. The law itself should be defined in the lede; defining it in terms of something derived from it does no one any good. It's been three and a half years and no one's done anything about it? 174.57.203.45 (talk) 19:12, 1 June 2012 (UTC)
replies: text:  Fixed now, I hope. --Jorge Stolfi (talk) 01:32, 9 May 2023 (UTC)
thread_title:  Lead 

message: The article currently states the following:
:That Zipfian distributions arise in randomly-generated texts with no linguistic structure suggests that in linguistic contexts, the law may be a statistical artifact.[2]
I don't get it, this sounds like patent non-sense. When one generates a random text, one must choose works with some random frequency or distribution.   What distribution is used?  The result of random generation  should show the same distribution as the random number generator: so if one generates random text using a Zipfian distribution, the result will be Zipfian. So this is a tautology. Surely something different is being implied in citation 2. However, this text, as written, is nonsense. linas (talk) 22:36, 14 March 2009 (UTC)
replies: text: Ah. The one sentence summary was misleading. I modified the article text so that is made sense.  The "random text" is actually a random string, chosen from an alphabet of N letters, with one of the letters being a blank space. The blank space acts as a word separator, and the letter frequencies are uniformly distributed. linas (talk) 22:59, 14 March 2009 (UTC)
replies: text: As the text is now, it's still (or again?) nonsense. It claims that the Zipfian distribution is a consequence of using letters to spell words. Just a few sentences above, the article says that words in the corpus are in Zipfian distribution. This has nothing to do with how words are encoded, and the distribution would exist regardless of what, if anything, is used to represent them. 193.77.151.137 (talk) 16:07, 5 April 2009 (UTC)
thread_title:  Nonsense? Err Nevermind 

message: The article originally stated,
In English, the frequencies of the approximately 1000 most-frequently-used words are approximately proportional to 1/{n^s} where s is just slightly more than one.[citation needed]
This is misleading, not just because the citation is missing, but also because it seems to me that English word frequencies may well follow the maximum entropy principle: their distribution maximises the entropy under the constraint that the probabilities decrease. Zipfs law, for any s>1, does not maximise the entropy, because it is possible to define distributions with probabilities that decrease even more slowly. For example, P(n) = 1/\log(n+1) - 1/\log(n+2) is a very slow one.
I therefore rewrote this bit somehwat.  —Preceding unsigned comment added by 131.111.20.201 (talk) 09:52, 1 May 2009 (UTC)
thread_title:  Frequency of words in English 

message: I think there is an error in the example log-log graph of Wikipedia word frequency vs. rank.  The two different lines of color cyan and blue cannot both represent 1/x^2, as they are labelled.  I suspect that one is 1/x^3.  Whoever has access to the code that generated the graph, please fix it!!!
--------------------  —Preceding unsigned comment added by 98.222.58.129 (talk) 04:45, 8 January 2010 (UTC)
replies: text: I agree the cyan plot is wrong but I don't think it can be 1/x3 since it nas the same slope as 1/x2.  It is more like 10110/x2.  I am going to ping the creator on this - it definitely needs fixing or removing. SpinningSpark 17:01, 4 June 2011 (UTC)

text: Actually, all the guide plots are up the creek: they should all be going through the point (1,1).  I suspect the author means \mathcal{O} f(x) rather than  \sim f(x) SpinningSpark 17:22, 4 June 2011 (UTC)
thread_title:  Problems 

message: I don't know how to add a citation needed link.
It's needed for the parenthetical, (although the law holds much more strongly to natural language than to random texts)
sbump (talk) 21:42, 19 March 2010 (UTC)
replies: text: type , it will look like this: . Cheers, — sligocki (talk) 03:21, 21 March 2010 (UTC)
thread_title:  citation needed 

message: Although the Shannon-Weaver entropy is listed in the table here, I think it is of fundamentally more importance and instructional value to list the REDUNDANCY which, in Zipf's law (s=1) is CONSTANT (and slightly less than 3).  Credit to Heinz von Foerster for deriving this from fundamental principles.
98.222.58.129 (talk) 14:04, 19 February 2010 (UTC)
thread_title:  Redundancy 

message: Dr. Ioannides has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:
We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of the article accordingly.
Dr. Ioannides has published scholarly research which seems to be relevant to this Wikipedia article:
*Reference : Yannis M. Ioannides & Spyros Skouras, 2009. "Gibrat's Law for (All) Cities: A Rejoinder," Discussion Papers Series, Department of Economics, Tufts University 0740, Department of Economics, Tufts University.
ExpertIdeasBot (talk) 04:53, 16 June 2016 (UTC)
replies: text: The article Power law seems like a better fit for everything that is described by a power law. 67.198.37.16 (talk) 18:22, 24 February 2020 (UTC)
thread_title:  Dr. Ioannides's comment on this article 

message: Hello fellow Wikipedians,
I have just modified  on Zipf's law. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive http://web.archive.org/web/20021018011011/http://planetmath.org:80/encyclopedia/ZipfsLaw.html to http://planetmath.org/encyclopedia/ZipfsLaw.html
When you have finished reviewing my changes, please set the checked parameter below to true or failed to let others know (documentation at ).
Cheers.—InternetArchiveBot (Report bug) 22:13, 20 July 2016 (UTC)
thread_title:  External links modified 

message: Both the phrases "Zipf distribution" and "Zipfian distribution" seem interchangeable but which is preferred?
80.194.75.37 (talk) 09:39, 27 January 2017 (UTC)
thread_title:  Zipf distribution or Zipfian distribution? 

message: The bit about information theory in the Applications section seems like it makes more sense if the word 'symbol' is changed to 'signal.' I don't know anything about the topic, though so I don't want to change it myself.
: Word 'symbol' is generally used in information theory - I have added '(event, signal)'. 188.146.143.213 (talk) 09:26, 27 April 2017 (UTC)
replies: text:  The section was removed pending clarification and sourcing; see below. --Jorge Stolfi (talk) 01:33, 9 May 2023 (UTC)
thread_title:  Should it say signal, not symbol? 

message: Several times in both this article and Zeta distribution is it stated that while they appear similar and are often referred to by the same name, they aren't related distributions. 
However, given the probability mass distributions of both distributions, I don't see why they aren't related.
The pmf of Zipf(s,N) is f_{Zipf}(k;s,N)=\frac{1/k^s}{\sum\limits_{n=1}^N (1/n^s)}, while the pmf of Zeta(s) is  f_{\zeta}(k;s) = \frac {1/l^s}{\zeta(s)} .
Now, the Riemann zeta function has a very simple form when  s > 1 , being  \zeta(s)= \sum\limits_{n=1}^\infty (1/n^s)  (from Riemann zeta function).
This tells me, at least, that the pmf of Zeta(s) is simply the pmf of of Zipf(s,N) in the limit of infinite elements,  \lim_{N\to\infty} f_{Zipf}(k;s,N) = f_{\zeta} .
Therefore, I am wondering if anyone has a source or (sketch of) proof that shows they aren't related in this manner, so we can cite it - or correct the article to imply that they are in fact directly related through the aforementioned limit. EpicScizor (talk) 13:56, 13 March 2021 (UTC)
thread_title:  Relation to the Zeta distribution 

message: The following unsourced and unclear section was deleted:
:Information theory
:In information theory, a symbol (event, signal) of probability p contains -\log_2(p) bits of information. Hence, Zipf's law for natural numbers: \Pr(x) \approx 1/x is equivalent with number x containing \log_2(x) bits of information. To add information from a symbol of probability p into information already stored in a natural number x, we should go to x' such that \log_2(x') \approx \log_2(x) + \log_2(1/p), or equivalently x' \approx x/p. For instance, in standard binary system we would have x' = 2x + s, what is optimal for \Pr(s=0) = \Pr(s=1) = 1/2 probability distribution. Using x' \approx x/p rule for a general probability distribution is the base of asymmetric numeral systems family of entropy coding methods used in data compression, whose state distribution is also governed by Zipf's law.
Is it original research? Jorge Stolfi (talk) 01:22, 9 May 2023 (UTC)
thread_title:  Removed unsourced section "Information theory" 

message: There is a reason the most popular youtube video on Zipfs law did not use a logarithmic graph...When you graph Zipfs law in Log scale,,,its just a linear graph...this is not understandable at a glance. A basic 1:1 graph is understandable without having to read the scale on axis (most don't even use axis its so easy). A exponential curve, sharply upwards at the end, is more understandable to the common people than anything with a logarithmic scale.
I ask we add 1 simple graph at the beginning so that a commoner can understand the curve easily. 
Non-logarithmic scales are readable instantly to everyone. (logarithmic are only readable to educated people, and even they have to check the axes) Diox8tony (talk) 17:34, 5 June 2023 (UTC)
thread_title:  Add a Non-Logarithmic graph (in the intro to the page preferably) 

message: Frequency as a function of frequency rank should be monotonically decreasing, which is not the case in the first plot of "word frequency vs frequency rank in War & Peace" (this plot --> https://en.wikipedia.org/wiki/Zipf's_law?/media/File:Zipf%27s_law_on_War_and_Peace.png#/media/File:Zipf's_law_on_War_and_Peace.png)
This plot isn't extracted from the source cited in caption (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4176592/pdf/nihms579165.pdf) but it is similar to other plots shown in this source. However all of the plots in this source are NOT monotonically decreasing, so it seems that those aren't a "Frequency as a function of frequency rank" plots.
I've never edited wikipedia before so I don't feel confident enough to propose the deletion of the plot. Qchenevier (talk) 15:13, 13 August 2023 (UTC)
thread_title:  The first plot seems to be wrong.  

message: I have removed the section on genlangs. The only citation was the original paper, which is neither notable or peer reviewed, and is of extremely low quality. Tristanjlroberts (talk) 16:42, 23 August 2023 (UTC)
thread_title:  Genlangs