Talk:Search engine indexing

message: ===Search engine sizes===
This comes from http://blog.searchenginewatch.com/blog/041111-084221, so I have not included it at the moment in the article, as I do not want to do anything illegal, and am not sure this is the best reference. The goal is to show the sizes, at least at some point in time, of the number of pages indexed, to help get a feel for the size. The understanding and reference to sizes in application is important to understand the technological challenge and the rationale behind the intense research in compression and forms of indexing and search engine architectures. Josh Froelich 16:44, 15 December 2006 (UTC)
replies: text: The size of an Internet search index isn't necessarily particularly useful information. The problem has never been how to store lots of entries in the index, as even simple solutions scale O(1) best-case and O(log(n)) worst case. In some space constrained world where you really had to save the bytes, you could probably fit a billion webpage reverse index in a few terabytes of data. If you crunch the numbers, you'll see that allows for a few kilobytes of information per document, which is plenty. The hard problem in search engine design is how to retreive the *relevant* entries. 176.10.248.194 (talk) 08:34, 13 September 2021 (UTC)
thread_title: Possible future content

message: This comes from http://blog.searchenginewatch.com/blog/041111-084221, so I have not included it at the moment in the article, as I do not want to do anything illegal, and am not sure this is the best reference. The goal is to show the sizes, at least at some point in time, of the number of pages indexed, to help get a feel for the size. The understanding and reference to sizes in application is important to understand the technological challenge and the rationale behind the intense research in compression and forms of indexing and search engine architectures. Josh Froelich 16:44, 15 December 2006 (UTC)
replies: text: The size of an Internet search index isn't necessarily particularly useful information. The problem has never been how to store lots of entries in the index, as even simple solutions scale O(1) best-case and O(log(n)) worst case. In some space constrained world where you really had to save the bytes, you could probably fit a billion webpage reverse index in a few terabytes of data. If you crunch the numbers, you'll see that allows for a few kilobytes of information per document, which is plenty. The hard problem in search engine design is how to retreive the *relevant* entries. 176.10.248.194 (talk) 08:34, 13 September 2021 (UTC)
thread_title: Search engine sizes

message: WikiPedia's Search option is powered by a search engine which involves an indexing process. The search engine technology is a program called Lucene (wiki), which is built into MediaWiki, the supporting software that WikiPedia uses. 
Some of these same concepts are employed by WikiPedia, and you can learn more about these by researching MediaWiki and the architecture of Lucene.
WikiPedia articles are stored in a database, which serves as the corpus. For indexing, primarily only the wikitext part of the article is considered, the other meta data is mostly ignored.  The wikitext is periodically indexed by the Lucene indexer, and this index is used when you click the search button on WikiPedia. From wikiehelp - Article does not appear when searching. The database for searching is updated approximately every 4-6 weeks. Changes do not appear there right away. Josh Froelich 02:34, 19 December 2006 (UTC)
thread_title: Notes on WikiPedia as an Example

message: Thinking about adding a future reading section or incorporating these sources into the article. Josh Froelich 22:38, 20 December 2006 (UTC)
thread_title: Further reading

message: This article would flow better if the sections were placed in the general order in which they are executed to create the inverted index (the final step). So something like:
# Document Parsing
# Forward keyword index
# Inverted Keyword index
With the smaller sections in between some of these. This would also involved changing the lead to include a statement of the goal of indexing: "Indexing aims to produce an map from unique keywords or terms to a list of documents containing that keyword. This data structure is referred to as an inverted keyword index."
Thoughts from anyone?
Drmadskills (talk) 18:16, 21 February 2008 (UTC)
thread_title:  Order of Sections 

message: This article seems too biased to Internet web search engines. I came here to find information on how full-text indexes work in general (hash tables?) and found lots of stuff about... identifying languages? meta tags? Web 3.0??  —Preceding unsigned comment added by 200.68.94.105 (talk) 19:06, 22 August 2008 (UTC)
thread_title:  Biased? 

message: One of the previous subject authors wrote: "Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, physics, and computer science." The word "informatics" is linked to another page titled Information technology. Note: Another page titled Informatics (academic field) exists. The Informatics (academic field) page specifies "Not to be confused with Information technology". This discussion requires a SME to identify 1. whether Index design incorporates informatics or information technology and 2. link the appropriate page to the designated word. araffals 17:11, 22 January 2013 (UTC)
thread_title: Suggested change

message: This article's lead section gives "search engine optimisation indexing" as a synonym for search engine indexing. Is this a mistake? Jarble (talk) 03:10, 29 September 2020 (UTC)
thread_title: "Search engine optimisation indexing"