# Presentation: 070757

## Using the Internet to Research Energy Statistics & Policy

- Dr. Mark Rodekohr
- Energy Information Administration

## Overview 

- Internet Search Techniques
- Search Engines
- How Search Engines Work
- Examples
- Conclusions

## The Problem

- The Internet contains anywhere from 300 million to 550 million or more publicly available documents - an amount doubling every 18 months.
- There is no Dewey decimal system or central "card catalog" for the Internet

## Two Types of Search Services 

- 'Directories' use trained professionals to classify useful Web sites into a hierarchical, subject-based structure. Yahoo is the best known and most used of these services. Directories are most useful when looking for information in clear categories.
- 'Search engines' work differently. Excite, AltaVista and Infoseek are some of the best known engines. They "index" (record by word) each word within all or parts of documents. When you pose a query to a search engine, it matches your query words against the records it has in its databases to present a listing of possible documents meeting your request.

## Before You Start

- Search engines are stupid, and can only give you what you ask for.
- Many engines will give thousands of returns.
- Poor queries return poor results; good queries may return great results.
- Most Internet searchers, perhaps including you, tend to use only one or two words in a query. Big mistake!

## Search Techniques

## Keywords – The Essence of the Search 

- The keywords in your queries will most often be nouns – and then likely no more than 6 or 8 of them.
- Always keep in mind the **who**, **what**, **where**, **when**, **how** and **why** in formulating your query.
- Never use articles, pronouns, conjunctions or prepositions – the connecting tissue in language – in your queries.

## Word Stemming and Use of Wildcards 

- Using AltaVista here are the document counts for the single and plural versions of **bird** (1,112,634) or **birds **(799,769)
- Wildcards can cause even more problems for example using **bird* **yields 1,834,510 returns.

## Finding the Right Level 

**THE MOST CRITICAL PROBLEM IN ALL QUERIES IS FINDING THE RIGHT LEVEL OF SPECIFICITY FOR THE SUBJECT QUERY TERM(S).  **Too broad a keyword specification, and too many results are returned; too narrow a specification, and too few are returned

## Finding the Right Level (continued)

**bird***	1,834,510
**falcon***	340,707
**peregrine falcon***	14,510

## Use of Phrases 

- Your most powerful keyword term is the phrase.  Phrases are combinations of words that must be found in the search documents in the EXACT order as shown.  You denote phrases within closed quotes (**“peregrine falcon*”**).  Some search services provide specific options for phrases, some do not allow them at all, but almost all will allow you to enter a phrase in quotes, ignoring the quotations if not supported.
- Always look for natural phrases in your query concepts – they are one of the most powerful weapons available.

## Boolean Basics 

-          **AND** – terms on both sides of this operator must be present somewhere in the document in order to be scored as a result
-          **OR** – terms on EITHER side of this operator are sufficient to be scored as a result
-          **AND NOT** – documents containing the term AFTER this operator are rejected from the results set
-          **NEAR** – similar to **AND**, only both terms have to be within a specified word distance from one another in order to be scored as a result
-          **BEFORE** – similar to **NEAR**, only the first (left-hand) term before this operator has to occur within a specified word distance _**before**_ the term on the right side of this operator in order for the source document to be scored as a result

## Boolean Basics (continued)

-          **AFTER** – similar to **NEAR**, only the first (left-hand) term before this operator has to occur within a specified word distance _**after**_ the term on the right side of this operator in order for the source document to be scored as a result
-          **Phrases** – combined words or terms that must appear directly _**adjacent**_ to one another and in the phrase order for the source document to be scored as a result
-          **Wildcards** (stemming) – beginning characters that must match the same beginning characters in a document’s words in order for it to be scored
**     ****Parentheses** – nested operators that are evaluated in an inside-out, then left-to-right order of precedence.

## Boolean Tips

**AND** should be your most frequently used Boolean operator.
- Use **OR** to string together synonyms; be careful about mixing it in with **AND** !.
- Use **NEAR** as an alternative to phrases and an improvement to **AND**, ***but only*** when you know the concepts are closely linked.
**AND NOT** is a powerful operator, use with care!  A single instance will cause a document to be excluded.
- Try to link three concepts together in your queries, joining with the **AND** operator.
**(“peregrine falcon*”) AND (“endangered species”) AND (city or cities)**

## Summary (1)

**1.**  Use nouns and objects as query keywords
**Ex: planet **or **planets**
- Actions (verbs), modifiers (adjectives, adverbs, predicate subjects), and conjunctions are either “thrown away” by the search engines or too variable to be useful
**2.**  Use 6 to 8 keywords in query
**Ex: new**, **planet**, **planets**, **discovery**, **solar**, **system**
- More keywords, chosen at the appropriate “level”, can reduce the universe of possible documents returned by 99% or more
**3.**  Truncate words to pick up singular and plural versions
**Ex: planet*** or **discover***
- Use asterisk wildcard.  The wildcard tells the search engine to match all characters after it, preserving keyword slots and increasing coverage by 50% or more
**4.**  Use synonyms via the **OR** operator
**Ex: discover* OR find**
- Cover the likely different ways a concept can be described; generally avoid **OR** in other cases

## Summary (2)

**5.**  Combine keywords into phrases where possible
**Ex: “solar system*”**
- Use quotes to denote phrases.  Phrases restrict results to EXACT matches; if combining terms is a natural marriage, narrows and targets results by many times
**6.**  Combine 2 to 3 “concepts” in query
**Ex: “solar system”**
**“****new planet*”**
**discover* OR find**
- Triangulating on multiple query concepts narrows and targets results, generally by more than 100-to-1
**7.**  Distinguish “concepts” with parentheses
**Ex: (“solar system”)**
**(“new planet*”)**
**(discover* OR find)**
- Nest single query “concepts” with parentheses.  (Overkill for now, but good practice when first learning.)  Simple way to ensure the search engines evaluate your query in the way you want, from left to right
**8.**  Order “concepts” with subject first
**Ex: (“new planet*”)**
**(discover* OR find)**
**(“solar system”)**
- Put main subject first.  Engines tend to rank documents more highly that match first terms or phrases evaluated

## Summary (3)

**9.**  Link “concepts” with the **AND** operator
**Ex: (“new planet*”) AND (discover* OR find) AND (“solar system”)**
**AND** glues the query together.  The resulting query is not overly complicated nor nested, and proper left-to-right evaluation order is ensured
**10.**  Issue query to full “Boolean” search engine or metasearcher
- Full-Boolean engines give you this control; metasearchers increase Web coverage by 3- to 4-fold

## Search Engine Basics

- Search engines use ‘spiders’ or ‘robots’ to go out and retrieve individual Web pages or documents, either because they’ve found them themselves, or because the Web site has asked to be listed.
- For example a examination of the EIA web logs indicates that many of these spiders visit our web site either very late at night or on the weekends.

## Search Engine Basics (2)

- Search engines tend to “index” (record by word) all of the terms on a given Web document.  Or they may index all of the terms within the first few sentences, the Web site title, or the document’s metatags.
*Precision*, *recall* and *coverage* are limiting factors for most search engines.  Precision measures how well the retrieved documents match the query; recall measures what fraction of relevant documents are retrieved

## Search Engine Basics (3)

- Coverage refers to what percentage of the potential universe of relevant documents is cataloged by the engine.
- Precision is a problem because of the high incidence of false positives.
- Coverage is a problem for all engines, with the largest ones only covering at most one third to one half of publicly-available documents

## Search Engine Coverage Stats

**Search Engine****	****% of all indexed pages****	****% that are dead links**
- Alta Vista	47%	2.5%
- Northern Light	39%	5.0%
- Inktomi	34%	Not Available
- Excite	17%	2.0%
- Lycos	16%	1.6%
- InfoSeek	14%	2.6%

## How Search Engines Rank Documents 

**Title** – an embedded description provided by the document designer; viewable in the titlebar (it is also used as the description of a newly created bookmark by most browsers)
-          **Description** – a type of metatag which provides a short, summary description provided by the document designer; not viewable on the actual page; this is frequently the description of the document shown on the documents listings by the search engines that use metatags
**Keywords** – another type of metatag consisting of a listing of keywords that the document designer wants search engines to use to identify the document.  These too, are not viewable on the actual page
**Body** – the actual, viewable content of the document

## How Search Engines Rank Documents (2)

- Search engines may index all or some of these content fields when storing a document on their databases.  (Over time, engines have tended to index fewer words and fields.)  Then, using proprietary algorithms that differ substantially from engine to engine, when a search query is evaluated by that engine its listing of document results is presented in order of ‘relevance.’

## How Search Engines Rank Documents (3)

**Order a keyword term appears**
**Frequency of keyword term**
**Occurrence of keyword in the title**
**Rare, or less frequent, keywords**
- There is one notable exceptions to these rules and that is the Google Search Engine which ranks documents by how long users (who enter a keyword similar to the one you used) spend on a particular web site.

## Top Search Engines

- AltaVista [http://www.altavista.com]
- Ask Jeeves [http://www.askjeeves.com]
- Direct Hit [http://www.directhit.com]
- Excite [http://www.excite.com]
- LookSmart [http://www.looksmart.com]
- Go [http://www.go.com]

- Google [http://www.google.com]
- HotBot [http://www.hotbot.com]
- Infoseek [http://www.infoseek.com]
- Inktomi [http://www.inktomi.com]
- LookSmart [http://www.looksmart.com]
- Lycos  [http://www.lycos.com]

## Top Search Engines (2)

- Magellan [http://magellan.excite.com]
- About.com [http://www.about.com]
- NetFind (AOL) [http://www.aol.com]
- Northern Light [http://www.northernlight.com/]

- Open Directory [http://dmoz.org]
- RealNames [http://www.realnames.com]
- Snap! [http://www.snap.com]
- WebCrawler [http://www.webcrawler.com]
- Yahoo [http://www.yahoo.com]

## Example of Search Engine Experiences

- PC Magazines John Dvorak used major search engines to look for a Hotel in Paris.
- His results are instructive of experiences with these engines.

## Excite

-  Excite found it under a list of hotels approved by something called Tools '96--a long gone trade show. Unfortunately, it was the wrong hotel, and the Excite search results were cluttered by bunch of obviously paid-for nonsense hits at the top of the list.

## HotBot

- HotBot returned a bunch of promotional garbage at the top of its list under the euphemism "search partners." That was followed by a list of some sort of directory hits that were totally useless and had to do with finding jobs in the hotel business

## Lycos

- Lycos had one of the more interesting results. It hit ten for ten with no duplication, and curiously all ten were quite different. It never found the home page for the hotel chain itself

## Alta Vista

- Alta Vista nailed the hotel six times, but started to list other Astor hotels, such as one in North Carolina, on search return number 7.

## Google

- Google also scored big, finding the hotel in nine out of its ten search results. Nevertheless, it didn't find the hotel chain home page, and many of the results were very offbeat.
- This is one of my favorite engines.
- It also shows the number and pages that are linked to the page in question.

## Northern Light

- Northern Light also failed to find the hotel home page, but hit the hotel appropriately in the first seven search results.
- This is one of the most highly rated engines by web users.

## Example: Gasoline Price Analysis

- This example compares the results of using different search engines to perform an analysis of gasoline prices.
- We start with the keyword “gasoline” followed by “gasoline + prices” followed by “gasoline + prices + analysis”.

## Google (http://google.com)

## Google Example

## Yahoo (http://yahoo.com)

## Yahoo Example

## Alta Vista (www.altavista.com)

## Alta Vista Example

## HotBot (www.hotbot.com)

## InfoSeek (infoseek.go.com)

## Webcrawler (webcrawler.com)

## Webcrawler Example

## Searching News Groups

## Conclusions

- Time spent structuring internet searches is time well spent.  There are to many links to view each one.
- Since internet content really starting in the mid-1990’s it is no substitute for going to a library for serious research, but it can provide a good start.
- Experiment with different search engines to find one that meets your needs and then get used to using it.
- Watch for new engines this is a area of rapid growth.