Search engine spider | Wikipedia audio article
Channel: wikipedia tts
Date: 10/07/2019

This is an audio version of the Wikipedia Article:
https://en.wikipedia.org/wiki/Web_crawler


00:01:29 1 Nomenclature
00:01:48 2 Overview
00:04:35 3 Crawling policy
00:05:10 3.1 Selection policy
00:09:38 3.1.1 Restricting followed links
00:10:53 3.1.2 URL normalization
00:11:37 3.1.3 Path-ascending crawling
00:12:25 3.1.4 Focused crawling
00:13:39 3.1.4.1 Academic-focused crawler
00:15:23 3.1.4.2 Semantic focused crawler
00:16:03 3.2 Re-visit policy
00:22:37 3.3 Politeness policy
00:26:02 3.4 Parallelization policy
00:26:40 4 Architectures
00:27:45 5 Security
00:28:38 6 Crawler identification
00:30:01 7 Crawling the deep web
00:31:27 7.1 Web crawler bias
00:31:49 8 Visual vs programmatic crawlers
00:33:06 9 Examples
00:35:11 9.1 Open-source crawlers
00:37:45 10 See also



Listening is a more natural way of learning, when compared to reading. Written language only began at around 3200 BC, but spoken language has existed long ago.

Learning by listening is a great way to:
- increases imagination and understanding
- improves your listening skills
- improves your own spoken accent
- learn while on the move
- reduce eye strain

Now learn the vast amount of general knowledge available on Wikipedia through audio (audio article). You could even learn subconsciously by playing the audio while you are sleeping! If you are planning to listen a lot, you could try using a bone conduction headphone, or a standard speaker instead of an earphone.

Listen on Google Assistant through Extra Audio:
https://assistant.google.com/services/invoke/uid/0000001a130b3f91
Other Wikipedia audio articles at:
https://www.youtube.com/results?search_query=wikipedia+tts
Upload your own Wikipedia articles through:
https://github.com/nodef/wikipedia-tts
Speaking Rate: 0.950176480094808
Voice name: en-US-Wavenet-D


"I cannot teach anybody anything, I can only make them think."
- Socrates


SUMMARY
=======
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering).
Web search engines and some other sites use Web crawling or spidering software to update their web content or indices of others sites' web content. Web crawlers copy pages for processing by a search engine which indexes the downloaded pages so users can search more efficiently.
Crawlers consume resources on visited systems and often visit sites without approval. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a robots.txt file can request bots to index only parts of a website, or nothing at all.
The number of Internet pages is extremely large; even the largest crawlers fall short of making a complete index. For this reason, search engines struggled to give relevant search results in the early years of the World Wide Web, before 2000. Today, relevant results are given almost instantly.
Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping (see also data-driven programming).