Talk:Letter frequency

message: The chart shown graphing letter frequency vs. letter is ordered alphabetically.  An additional chart ordering the vertical bars by frequency (rather than alphabetically) would enhance the presentation.  
:I generated such a frequency-ordered chart on my Windows system using the Excel spreadsheet chart facility.  I have not tried to add the result to the Wiki article because it's relatively ugly and because I couldn't figure out how to convert it to a .png file.
I've found an ordered letter frequency of the english language in this page:
http://www.csm.astate.edu/~rossa/datasec/frequency.html
The source of the table is: H. Beker and F. Piper, Cipher Systems, Wiley-Interscience, 1982.
I don't know if it would be ok to put it here. 
----
How about using this source: 
Case-sensitive letter and bigram frequency counts from large-scale English corpora. MN Jones, DJK Mewhort - Behavior Research Methods, Instruments, & Computers, 2004
A link to it can be found here 
--Zip123 (talk) 16:37, 23 October 2008 (UTC)
thread_title:  Chart ordered by frequency would be helpful 

message: In the book The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography by Simon Singh, I found the following table with a caption that reads:
:This table of relative frequencies is based on passages taken from newspapers and novels, and the total sample was 100,362 alphabetic characters. The table was compiled by H. Beker and F. Piper, and originally published in Cipher Systems: The Protection Of Communication.
Note that the values below add to 100.3 due to rounding.
Letter	Percentage	Letter	Percentage
a     	8.2       	n     	6.7
b     	1.5       	o     	7.5
c     	2.8       	p     	1.9
d     	4.3       	q     	0.1
e     	12.7      	r     	6.0
f     	2.2       	s     	6.3
g     	2.0       	t     	9.1
h     	6.1       	u     	2.8
i     	7.0       	v     	1.0
j     	0.2       	w     	2.4
k     	0.8       	x     	0.2
l     	4.0       	y     	2.0
m     	2.4       	z     	0.1
The following table sorts the values given above in order of letter frequency.
Letter	Percentage	Letter	Percentage
e     	12.7      	m     	2.4
t     	9.1       	w     	2.4
a     	8.2       	f     	2.2
o     	7.5       	g     	2.0
i     	7.0       	y     	2.0
n     	6.7       	p     	1.9
s     	6.3       	b     	1.5
h     	6.1       	v     	1.0
r     	6.0       	k     	0.8
d     	4.3       	j     	0.2
l     	4.0       	x     	0.2
c     	2.8       	q     	0.1
u     	2.8       	z     	0.1
:I took a little class on cryptology once, and The Code Book and Cryptological Mathematics were our textbooks.  I'm pretty sure they have the same data, but in The Code Book it's rounded.  --Ravi12346 19:40, 30 July 2006 (UTC)
thread_title:  Statistics from a larger sample size 

message: sth is a surprise in a list of high-frequency trigrams. On its own it's an abbreviation of south, and I can think of a few words containing it, but not enough to account for its listing here.  Can anyone tell me what it is I haven't thought of? 
I grepped a dictionary and came up with 414 results... admittedly, almost all of them you wouldn't use in conversation (try "somesthetic" and "chromesthesia"), but there are a couple like 'firsthand' and 'guesthouse' that aren't so outlandish.
is as has was / this the that there they
Trigraphs ignoring spaces may not be of great practical use though.
Uldoon 10:33, 10 March 2006 (UTC)
thread_title: Query

message: contains a lot more "z"s than British English. 218.102.218.250 03:02, 5 April 2006 (UTC)
thread_title: American English

message: So...H and D appear more than L, but the "gimme" letters in the last round of Wheel of Fortune are RSTLNE.  That should appear somewhere towards the bottom of this article.  --JD79 19:20, 26 January 2007 (UTC)
replies: text: Pure speculation here, but (1) I think those are the gimme letters because people had gotten the idea that they were a "good set" and guessed the exact same letters all the time, and the producers probably wanted to mix it up; and (2) if you've already guessed S and T, you can probably infer the locations of H's, as well as the likelihood of the last letter's being D (the past tense prefix no doubt being the reason why D is so common). --Mr. A. 04:21, 27 January 2007 (UTC)
thread_title: Wheel of Fortune

message: From the article:
:"The frequency of letters in text messages has often been studied for use in cryptography..."
In the UK at least, the term "text messages" (SMS messages TO SOME DUMBASSES), and readers could hardly understand it to mean anything else. However, from the context it doesn't seem very likely that this was the intended meaning, at least not exclusively. If not then I suggest "text messages" is replaced by just "text". I won't change it just yet in case anyone has a good reason why it should read as it does. Matt 20:36, 21 April 2007 (UTC)
replies: text: I think you're absolutely right, and I've gone ahead and made the change. --Mr. A. 10:49, 24 April 2007 (UTC)
thread_title:  "Text messages"? 

message: I've removed the Hemingway vs Faulkner section in Letter frequencies, again, because, aside from the quote about conventional use of punctuation, it's unreferenced, and may be original research. It's also not really germane to the article; showing that two particular works have different letter frequencies doesn't really say anything about those writers' styles - they may have chosen character or place names which account for the entire discrepancy. An analysis, to be meaningful, would need to show similarities across all of an author's works for several authors, and consistent differences between authors. If there is any published literature on the subject, then it would be appropriate to  summarize it and include it in this article under a section heading such as letter frequency variation and authorial style. Argyriou (talk) 19:39, 23 July 2007 (UTC)
thread_title:  Hemingway and Faulkner 

message: I recall having read a book that used an alphabet ordered by frequency, beginning with the letters above, that was useful in solving substitution ciphers.  Searching for "etaonrish" on Google gets quite a few results, too, so I wonder why this order isn't discussed in this article. B7T (talk) 04:11, 17 April 2008 (UTC)
replies: text: This is the list that I mention at the end of the Dispute section above. I'd be interested in knowing where this list of frequencies came from. --Geenius at Wrok (talk) 10:41, 19 April 2008 (UTC)
replies: text: If the title in that section is correct, a query on Amazon.com suggest that the book in question may be a children's book, The First Book of Codes and Ciphers by Sam and Beryl Epstein, illustrated by Laszlo Roth, published in 1956, if anyone wants to try to find a copy to verify it; I'm fairly certain that I first heard of this sequence in a different book for young people.  A search for "etaonrish" on Amazon.com yielded results of its mention in five other books in the context of ciphers or cryptography, none of which are children's books (although one is about activities to do with children). B7T (talk) 19:44, 24 April 2008 (UTC)
replies: text:  If you ever figure it out, perhaps it would also be useful in the ETAOIN SHRDLU article. --68.0.124.33 (talk) 17:21, 9 March 2010 (UTC)
replies: text: The full sequence of the letters is ETAONRISHDLFCMUGYPWBVKXJQZ and can be found in a children's book by the name of "Alvin's Secret Code" 1963 by Clifford B. Hicks.  — Preceding unsigned comment added by 204.44.186.204 (talk) 17:37, 2 March 2015 (UTC)

text: This sequence is also mentioned in the children's book "Codes and Secret Writing" by Herbert Zim (1971). According to Zim, "Many studies of the frequency of letters in the English language have been made. Samuel Morse made such a study when he made his Morse Code. Edgar Allan Poe did too. Dozens of other studies are all in reasonably close agreement. The printer who sets up type knows this fact about our alphabet and uses it every day. The case in which he keeps his type is divided up into compartments. The letters used most get the largest boxes and those that are easiest to reach." (p71) Dabley2 (talk) 13:15, 2 May 2017 (UTC)
thread_title:  ETAONRISH 

message: Shouldn't this article be moved to Letter frequency per WP:SINGULAR? Leon math (talk) 21:47, 4 January 2009 (UTC)
thread_title:  Move to "Letter frequency"? 

message: Why is the ë 0.000% instead of 0?  Is there only one word in their entire language with it?  And is it only used in certain times of year? (Noël)  Is Noël the only example?  Is it 0.000% because the percentage is less than 0.000% but greater than 0?
Uber-Awesomeness (talk) 19:49, 10 February 2009 (UTC)
replies: text:  Off top of my head, I can think of "canoë", "continguë", "ambiguë", "noël", "exiguë", "ciguë", "aiguë", and "Israël", "Staël", "Saint-Saëns". We also use "ü" in "capharnaüm", "Saül", "Esaü". And of course, we use lots of "ï" as in "ambiguïté", "exiguïté", "égoïste", "aïeul", "glaïeul", "haïr", "maïs", "coïncider", "inouï", etc...  —Preceding unsigned comment added by 84.72.92.4 (talk) 00:58, 16 April 2009 (UTC)
thread_title:  French ë? 

message: Why not to include in statistics English diacritics such as those within English terms with diacritical marks article.  — Preceding unsigned comment added by 86.75.160.141 (talk) 22:23, 27 October 2012 (UTC)
replies: text: I added those that I could find examples for. It would be good to regenerate the stats using a better dataset. I've seen it spelled "naïve" so many times that I think if you're an adult English reader you're almost required to know it and the frequency table doesn't reflect that reality. Akeosnhaoe (talk) 04:48, 19 November 2020 (UTC)
thread_title: Diactric in english language

message: I must question whether Toki Pona — a constructed language of recent origin, with only a few dozen users, and intentionally designed not to be of general use as an international auxiliary language — is sufficiently notable to merit inclusion in the "Relative frequencies of letters in other languages" table.  Comments?  Richwales (talk · contribs) 06:11, 22 December 2010 
(UTC)
There is absolutely no excuse for the obscure Toki Pona to be in this article. The language itself is not really notable enough to deserve an article (I'm shocked it has one). If the goal is to include a made-up language, I'd suggest Pig Latin. If the goal is to include a language with a very different distribution, I'd suggest Hawaiian. If it weren't such a pain to edit tables on Wikipedia, I would have ripped Toki Pona out already. Perhaps someone knows of an easier way to edit tables. RoyLeban (talk) 22:06, 9 September 2011 (UTC)
thread_title:  Toki Pona 

message: Is there a source for letter frequency by language more generally? (That is, for languages not tabulated here.) It seems to me it would be useful to mention or link to, for people wanting more information.  TREKphiler any time you're ready, Uhura  16:29, 13 April 2011 (UTC)
thread_title:  Universal translator? 

message: Entirely original research, but if anyone is seeking independent confirmation of (approximate) letter frequencies, I'd recommend taking a look at their computer keyboard. My netbook - a few months old - had a matt surface on the keys. The well-used ones are now glossy... AndyTheGrump (talk) 03:15, 8 June 2011 (UTC)
thread_title:  Strictly [[WP:OR]] 

message: At 7 paragraphs, the intro for this article appears to be excessive.  Rather than deleting any content, perhaps some of the material could be moved into its own section or merged with others.  I'd like to add a cleanup tag, called , to resolve this problem ... unless there are objections, or a specific reason for the verbosity? — VoxLuna ☾ orbitland  22:11, 9 November 2011 (UTC)
thread_title:  Introduction 

message: The sum of the frequency of the letters for the languages English, French, German, Spanish, Portuguese, Esperanto, Italian, Turkish, Swedish, Polish, Dutch, Danish, Icelandic, and Finnish are currently 100.00%, 99.16%, 102.36%, 104.47%, 104.09%, 99.99%, 101.14%, 94.01%, 100.00%, 108.01%, 101.59%, 100.00%, 100.00%, and 100.00%.  Whilst I understand that frequencies may not always add up to 100%, I cannot imagine why they should ever exceed 100%.  I cannot avoid the conclusion that some letters are being counted twice.  — Preceding unsigned comment added by 2.104.4.142 (talk) 14:08, 15 February 2014 (UTC)
replies: text: It's possible that the diacritics are being counted twice - checking a reference at random, the source for the Portuguese distribution differs slightly (it totals 100.01% versus this article's 104.09% - the .01% is presumably just a rounding error, although H, W and X have different values) and it says absolutely nothing about diacritics, so perhaps somebody just added them afterwards from a different source. --McGeddon (talk) 09:58, 21 February 2014 (UTC)
replies: text: Looks like these edits from May/June 2013 might be the culprit - an IP editor added more diacritic rows and updated the stats for every language, somehow bumping them up to three decimal places of accuracy despite the sources only using two. Assuming good faith, this could have been somebody running their own analysis of corpus texts and getting a finer level of accuracy, but it's left us with a lot of data that doesn't match its sources, and apparently doesn't quite add up. --McGeddon (talk) 10:05, 21 February 2014 (UTC)
thread_title:  Tables do not add up to 100% 

message: On the 5th through the 8th of July 2014 three IP-numbers: , ,  made unsourced changes in the article, mainly in the table with data from other languages, amongst them Polish, Turkish and Dutch data. For some of the data I can not read or understand the original language, from what I can read I discovered (intentional) errors. Today I'll revert only those from the Dutch language to the source data. --VanBuren (talk) 15:05, 16 July 2015 (UTC)
replies: text: Good catch. As the tag at the top of the table says, the numbers do not match the supposed sources. This is noticeable with the number of decimal places in the Dutch column, which are almost entirely given to only two decimal places in the cited source. We should change everything to match the sources. Sminthopsis84 (talk) 15:15, 16 July 2015 (UTC)
thread_title:  Apparently corrupted data 

message: Why 25? This number seems a bit random. All aforementioned languages have 26 or more letters, the exception being Italian with only 21 official letters. 25 can't be the average either [French (26), German (26), Spanish (27), Portuguese (26), Esperanto (28), Italian (21), Turkish (29), Swedish (29), Polish (32), Dutch (26), Danish (29), Icelandic (32), Finnish (28), Czech (42)]. --147.142.185.206 (talk) 11:12, 5 March 2016 (UTC)
thread_title:  All of these languages use a similar 25+ character alphabet. 

message: This article only addresses the frequency of letters without distinction of whether they are occur in lowercase or uppercase in the corps of texts analyzed. For instance, in the books of Project Gutenberg (mentioned here), there certainly are enough words with uppercase letters - mainly at the beginning of a sentence - to make that distinction meaningful. In some applications, (e.g. text compression) this information is indeed valuable. Since the article is titled "Letter frequency" and uppercase letters are different from lowercase letters, I'd argue that addressing this point belongs into this article.Havajsky (talk) 19:57, 30 March 2017 (UTC)
replies: text: Michael N. Jones and D. J. K. Mewhort, “Case-sensitive letter and bigram frequency counts from large-scale English corpora”  (Behavior Research Methods, Instruments, & Computers, 2004, 36 (3), 388–396) is very informative. (Notably, uppercase E is relatively rare, with a frequency rank of 11th out of 26 uppercase letters, while lowercase e is by far the most frequent.) Kanou-h (talk) 09:56, 15 November 2023 (UTC)
thread_title:  Uppercase/lowercase Frequency 

message: Hello fellow Wikipedians,
I have just modified one external link on Letter frequency. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive https://web.archive.org/web/20070930194046/http://gpl.insa-lyon.fr/Dvorak-Fr/CorpusDeThomasTemp%C3%A9 to http://gpl.insa-lyon.fr/Dvorak-Fr/CorpusDeThomasTemp%C3%A9
When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.
Cheers.—InternetArchiveBot (Report bug) 17:17, 14 May 2017 (UTC)
thread_title:  External links modified 

message: Hello fellow Wikipedians,
I have just modified 2 external links on Letter frequency. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive https://web.archive.org/web/20090922080351/http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml to http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml
*Added archive https://web.archive.org/web/20070402181401/http://pages.central.edu/emp/LintonT/ to http://pages.central.edu/emp/LintonT/
When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.
Cheers.—InternetArchiveBot (Report bug) 20:07, 4 June 2017 (UTC)
thread_title:  External links modified 

message: Though they are not Added to the Dutch alphabet é and ë are regularly (and ï less*) used in writing Dutch. één for exempel in Dutch means one while een means a. This letter is also used to accentuate a letter e so the pronaunciation is clear. The letter ë is even a lot more frequently used in Dutch. For exempel in the words industriële, accentuële or artificiële. It's also frequently used for geogrphical locations like Belgium (België), Asia (Azië), oceania (Ocenanië) Tasmania (Tasmanië), Transylvania (Transsylvanië)... (an exeample of ï: maïs or pinguïn*) This explained: why are these letters not included in the list? Falco iron (talk) 11:05, 10 September 2017 (UTC)
thread_title:  Dutch é and ë 

message: As mentioned above, accented letters do occur in Dutch. I went to the source and found a more recent one. They obviously removed accents before determining frequency, as the word "één" (1) does not occur on the list of 200 most frequent Dutch words from the same corpus, while twee, drie and vier (2, 3 and 4) all do occur on the list.
Unfortunately I have not been able to find a source of better data.
BGPexpert (talk) 13:13, 19 February 2024 (UTC)
thread_title:  Dutch frequencies of accented letters are incorrect 

message: I think it would be interesting to explain that the order of Morse letters by length, e it san hurdm wgvfbk pxz jyq, puts o, l, and c in an odd place (far towards the end despite of their high frequency) because the current International Morse code is actually a modification of the original version, where this would look like e t i an ms dfru ghkw bqvxyz jp.
I added a footnote explaining this but then was told that, without references, this might be considered original research so I undid it.
However I'm opening this discussion to see if someone can add a reference to said fact or if this footnote can be safely added as is (since this information can be almost directly obtained from the image at American Morse code) without any reference.
—Cousteau (talk) 00:33, 4 December 2017 (UTC)
thread_title:  Morse 

message: The page reads "The first method, used in the chart below, is to count letter frequency in root words of a dictionary," so I take it that this is referring to Pavel Mička's list. However, the data for the chart on the right from Nandhp with the caption "Relative frequencies of letters in text" has exactly the same numbers. So which is it? Are these numbers for the roots or for average text?
195.191.126.1 (talk) 13:27, 29 December 2017 (UTC)
thread_title:  Letter Frequency in Roots vs Text 

message: Which is right? vbgkjq or vbgkqj?99.10.204.30 (talk) 14:09, 12 July 2018 (UTC)
thread_title:  Vbgkjq 

message: It seems the frequencies for the letters 'w' and 'k' for English have been changed and the frequencies no longer adds up to 100%. It's now 106%. 'w' previously was 2.36 (now 5.37) and 'k' was 0.772 (now 3.872). No sure who changed it or why, but if the new frequencies of 'k' and 'w' are more accurate, the other frequencies need to be adjusted to add up to 100% again.  — Preceding unsigned comment added by Karl Napf der Abwaschbare (talk • contribs) 04:48, 8 December 2018 (UTC)
thread_title:  English frequency column now add up to 106% 

message: Accents are used in English occasionally. Most commonly, it appears in names, but there are also a few words that can optionally have accents or other marks, such as résumé (with one or both accents), naïve, façade, and a few others. Should the "frequency by language" chart be updated with very low nonzero frequencies for English, or does it round to zero? HotdogPi 13:26, 27 June 2019 (UTC)
replies: text: I very much agree with HotdogPi (and many others) regarding the flaws of this article as stands. It appears the biggest problem is a lack of citable resources with which to produce more informative infographics. I still propose that anyone interested in this topic find some resources on frequency analyses that rank the frequency of all unicode characters on some List_of_text_corpora. Original research is not permitted on Wikipedia, however, anyone with sufficient skills and interest could also just publish their own research, (so as to meet the requirements of citability/reliability, of course). See M. Jankowska, V. Kešelj and E. Milios, "Relative N-gram signatures: Document visualization at the level of character N-grams," 2012 IEEE Conference on Visual Analytics Science and Technology (VAST), Seattle, WA, 2012, pp. 103-112, doi: 10.1109/VAST.2012.6400484. as a potential starting point for further research? BlackPlatinumChowChow (talk) 15:52, 22 May 2020 (UTC)
thread_title:  Are accented letters in English actually 0.000%? 

message: The percentages in the table 'Relative frequency as the first letter of an English word' add up to 90%.
Should the percentages' sum be 100%?
Thank you,
2604:6000:7750:B800:ACBA:D86:9628:57D (talk) 05:33, 29 November 2020 (UTC)
thread_title:  'Relative frequency as the first letter of an English word' add up to 90% 

message: Why are the etaoin shrdlu equivalents listed at two groups of five letters, when etaoin shrdlu is two groups of six? If there is a reason for this, wouldn't it be useful to include this in the article? 84.92.90.18 (talk) 10:57, 27 May 2021 (UTC)
thread_title:  Etaoin Shrdlu equivalents 

message: At present among the tables in this article, some use A B C D etc., others use a b c d etc. Besides being inconsistent in style, this may also mislead some readers about what is being counted: only upper case? only lower case? both actually?
I would like to update each table to follow the Aa Bb Cc Dd etc. paradigm seen in other articles, such as List of Latin-script letters and List of Cyrillic letters.
- DuckMaestro (talk) 18:37, 7 August 2022 (UTC)
thread_title:  Inconsistent Uppercase/Lowercase 

message: Summing up the letter frequency for Hungarian letters gives 117,531% instead of expected 100%. Milan Berta 07:55, 25 March 2023 (UTC)
replies: text: Dexxor (talk) 12:23, 25 March 2023 (UTC)
thread_title:  The sum of the freq % is not 100 for Hungarian letters 

message: Considering that k is a Foreign letter in French, appearing only in loanwords? This link reports an even higher frequency for k, wow! 129.104.241.242 (talk) 04:53, 3 June 2024 (UTC)
thread_title:  How could French k share the same frequency (0.074%) as the English z?