# Presentation: 584948

## Search Secrets: Technology Intelligence (Finding What You Want)

- Larry G.Hull
- May 20, 2004

## Presentation Goals

- Common Search Mistakes
- Basic Search Techniques
- Major Search Engines
- Google Secrets
- Advanced Search Secrets
- Google Advanced Operators and Information
- Google Resources
- Other Technology Intelligence Resources
- Search Engines’ Future Goals

## Biggest Search Mistakes

- Typing URLs into the wrong box

## Top Search Terms - 2:00 May 13 

- Kelly Blue Book
- Paris Hilton / Pamela Anderson
- Mapquest
- Google
- Games
- Yahoo
- Jokes

- Health
- Thong / Thongs
- eBay
- Dictionary
- Lyrics
- Hotmail
- Friends Finale
- Ask Jeeves

## What Did You Notice? 

- Kelly Blue Book
- Games
- Mapquest
- Google
- Paris Hilton / Paris Hilton Video
- Yahoo
- Jokes

- Health
- Thong / Thongs
- eBay
- Dictionary
- Pamela Anderson
- Hotmail
- Friends Finale
- Ask Jeeves

## My Point 

- Kelly Blue Book
- Games
- Mapquest
- Google
- Paris Hilton / Paris Hilton Video
- Yahoo
- Jokes

- Health
- Thong / Thongs
- eBay
- Dictionary
- Pamela Anderson
- Hotmail
- Friends Finale
- Ask Jeeves

## Biggest Search Mistake

- Typing URLs into the wrong box
- Search terms entered in the address field
- Addresses entered in the search field
- Source: _[wordtracker.com](http://wordtracker.com/)_
- Today’s top search terms
- Shown as moving list at top of page
- Download using “Send words to a friend” link
- People continue to type addresses as search terms, even after the mistake has been pointed out to them

## Biggest Search Mistakes

- Typing URLs into the wrong box
- Using the wrong tool at the wrong time

## The Second Biggest Mistake

- Using a directory instead of a search engine ... and then not understanding why you can’t find anything!
- Would you look up the definition of a word in the phone book?
- Directories are human-compiled and have a small number of pages in their databases (usually in the low millions)  e.g. _[MSN](http://www.msn.com/)__ or Yahoo_
- Search engines are machine-compiled and have a **very large** number of pages in their databases (usually in the hundreds of millions or even billions)

## Directories vs. Search Engines

- Directories are **great** for “telephone book” searches
- What is the Web page address for company A, organization B, or entity C?
- Who makes product X?
- Where is a list of Web pages for topic Z?
- Search engines are great for “encyclopedia” or “dictionary” searches
- What is Z?
- Three parts to a search engine
- Spider or bot (sometimes called a crawler)
- Index or catalog
- Front end or retrieval page

## Directories vs. Search Engines

- Directories may grow to where they effectively become a small search engine or may explicitly decide to transform themselves into one, e.g., Yahoo
- Search engines may divide into topic or functionality oriented subsets that resemble directories, e.g., _[Google](http://www.google.com/)_
- Some offer unique front ends, e.g., Ask Jeeves and KartOO (offering multisearch and a visual display)
- Competition is fierce

## People Use Directories -- Why?

- Search engines have more, far more, pages in their databases than do directories
- Why do people predominantly use directories?
- They know how to use a directory
- No one has ever shown them how to use a search engine!
- Note: This is also true for a search box or page at a Web site.
- People will say they prefer to use a site’s search facility.
- Usability tests show most use the site map if one exists.

## Biggest Search Mistakes

- Typing URLs into the wrong box
- Using the wrong tool at the wrong time
- Not knowing when and how to use directories or search engines to actually find information

## Search Basics

- Be specific ... because if you aren’t specific, you’ll end up with a bunch of garbage!
- minimum of three to five specific terms, e.g.,
-          ocean biology and biogeochemistry data mining
-      and is a “stop word”, i.e., an extremely common word that is frequently discarded to speed up searches

## Search Basics

- Be specific ... because if you aren’t specific, you’ll end up with a bunch of garbage!
- Use quotes to search for phrases
-          “ocean biology” biogeochemistry “data mining”

## Search Basics

- Be specific ... because if you aren’t specific, you’ll end up with a bunch of garbage!
- Use quotes to search for phrases
- Use the + sign to require
-  “ocean biology” biogeochemistry +“data mining”

## Search Basics

- Be specific ... because if you aren’t specific, you’ll end up with a bunch of garbage!
- Use quotes to search for phrases.
- Use the + sign to require
- Use the - sign to exclude
- “ocean biology” biogeochemistry +“data mining” -solicitation

## Search Basics

- Be specific ... because if you aren’t specific, you’ll end up with a bunch of garbage!
- Use quotes to search for phrases.
- Use the + sign to require.
- Use the - sign to exclude.
- Combine symbols as often as possible (see rule #1)
- +“ocean biology” biogeochemistry +“data mining” -solicitation

## Major Search Engines

- Google           http://www.google.com
- Well-deserved reputation as top choice for searching web
- Crawler-based service provides comprehensive coverage and great relevancy
- Recommended as a first stop
- Yahoo            http://www.yahoo.com
- Web's oldest (1994) directory
- Shift in October 2002 to crawler-based listings from Google
- Second shift in February 2004 to its own search technology

## Major Search Engines - 2

- Ask Jeeves     http://www.ask.com/
- Ask Jeeves initially gained fame in 1998 and 1999 as being the "natural language" search engine
- Behind the scenes, editors monitored search logs and went out onto the web to located what seemed to be the best sites
- Today, Ask Jeeves depends on crawler-based technology
- HotBot                http://www.hotbot.com
- HotBot provides easy access to the web's three major crawler-based search engines: Yahoo, Google and Teoma (pronounced tay-o-ma and owned by Ask Jeeves)
-  Unlike a meta search engine, it cannot blend the results from all of these crawlers together

## Meta Search Engines

-          Unlike search engines, metacrawlers don't crawl the web to build listings. Instead, they allow searches to be sent to several search engines all at once. The results are then blended together onto one page.
- Dogpile                http://www.dogpile.com
- Searches a customizable list of search engines, directories and specialty search sites
- Displays results from each search engine individually

## Meta Search Engines - 2

- MetaCrawler       http://www.metacrawler.com/
- One of the oldest meta search services,
- MetaCrawler began in July 1995 at the University of Washington
- KartOO              http://www.kartoo.com
- Shows search results using a visual display
- Sites are interconnected by keywords
- Looking for search engines?
- http://searchenginewatch.com/links/

## Google Secrets

- It’s easy if you remember a few, simple commands
- Title searches
- Search for pages that have a particular word or phrase in their title
-                             **intitle:*****terms***
- Examples
**                            ****intitle:moon**
**                            ****intitle:”Moon Landing”**

## Google Secrets - 2

- Site searches (used with another term)
**                 *****term(s)***** site:*****domainname***
- Limit search to only pages within a specific site or domain
**                            ****apollo site:nasa.gov**
**                            ****apollo +site:gov**
- Exclude pages from a specific site or domain
**                            ****apollo -site:nasa.gov**
**                            ****apollo -site:gov**

## Google Secrets - 3

- URL searches
- Search for pages that have a particular word or phrase in their URL
- Works great when you can only remember one word in a long URL
**                            ****inurl:*****url***
**                            ****inurl:apollo**
- Example of “real” search
** ****+inurl:apollo +moon -“John Young” +site:nasa.gov**
- Note: does not mean Young won’t appear in the site

## Google Secrets - 4

- Related searches
- Search for pages that are similar to another page
**                           ****related:*****url***
- Related search to find pages similar to gsfc.nasa.gov
**                            ****related:gsfc.nasa.com**
- Typical result will be *servername*.gsfc.nasa.gov
- Note: related search results equivalent to URL “similar pages”

## Advanced Search Secrets

- How does search really work in search engines?
- Only the insiders know for sure
- People have made some shrewd guesses
- Number of keywords appearing on page scores “X”
- Keyword adjacency (closeness) on page scores “Y”
- Number of appearances of a keyword is given a weight and composite of weights scores “Z”
- Search engine uses X, Y, Z, and other secret variables to come up with a score for each page; applies a page rank, and displays to top n pages m at a time
- What is a page rank?
-     Note: “PageRank” is Google’s secret algorithm

## Page Rank (Digression)

- Premise: importance of a research paper can be judged by the number of citations
- Analogy: importance of a Web page can be judged by the number of links pointing to it from other pages
- Or, mathematically, where PR = page rank and C = citations (links)
-         d is a damping factor in the range of 0 < d < 1, perhaps 0.8

## Page Rank (continued)

- The page rank of a Web page is the sum of the page ranks of all the pages linking to it divided by the number of links on each of those pages
- A page with many links to it from other pages is judged to be more important than a page with only a few links from other pages
- A page with only a few links to other pages is deemed to be more important than a page with links to lots of other pages, e.g., a portal

## Boolean Operators 

- Search engines use AND as their Boolean default
- Automatically look for pages that contain ALL keywords
-        Google: The "AND" operator is unnecessary -- we include all search terms by default.
- One reason to use phases, e.g., “data mining”
- Some searches need a Boolean OR
- OR operator is (almost) *always* in all caps
- Goes between keywords
- Often may be (should be) used with parentheses
-          ocean biology **and** biogeochemistry data mining
-          “ocean biology” biogeochemistry “data mining”
-          (“ocean biology” OR biogeochemistry) “data mining”

## More Tips

- Capitalization may not matter (except Boolean OR)
- Number of keywords may be limited
- Don’t use stop words: a, about, an, and, are, as, at, be, by, from, how, i, in, is, it, of, on, or, that, the, this, to, we, what, when, where, which, with
-     Note: try using a + to search for a stop word, +we
- Use wildcard if supported by search engine
- Wildcards are characters, usually asterisks (*), that represent other characters
- Google offers full-word wildcards, **it’s +a * world**
- Some search engines support a technique called “stemming”, some* matches sometime, somewhere, ...

## More Tips 2

- Order of the search keywords matters
- Number of results (hits) will probably be the same
- Ordering of the hits, particularly the top hits, will vary
- Some search engines offer advance search (page and/or advanced search operators)
- Search specific parts of Web pages
- Search for specific types of information
- We’ve seen some of Google’s advanced operators
- Query modifiers, e.g.,  intitle:, inurl:, site:
- Alternative query types, e.g., related:
- Information types, e.g., define:

## Google Advanced Operators

**filetype:**
- Results restricted to files with a particular extension ".doc”, “.xls”, “.ppt”, etc
- Shows only files created with the corresponding program
- No space between filetype: and the file extension
- “dot” in the file extension is optional
**             “****data mining” ****filetype:pdf**
**daterange****:**
- Date or range of dates page was indexed
- Works only with Julian dates
**             “****data mining” ****daterange:2452401-2452766**

## Google Advanced Operators - 2

**inanchor:**
- Results restricted to text in a page’s anchors (links)
- No space between inanchor: and the first word
- Phases (in quotes) do work
-    <p>Please <a href=“guestbook.html”>sign our guestbook</a></p>
-    **terms**** inanchor:guestbook**
**inte****x****t:**
- Searches body text and only body text
- No space between intext: and the first word or phase
**          ****terms ****intext:“Goddard Space Flight Center”**

## Google Information

- Phonebook:, three versions
**phonebook:** searches the entire Google phonebook
**rphonebook:** searches residential listings only
**bphonebook:** searches business listings only
- parameters
- first name (or first initial), last name, city (state is optional)
- first name (or first initial), last name, state
- first name (or first initial), last name, area code
- first name (or first initial), last name, zip code
- phone number, including area code
- last name, city, state
- last name, zip code
-                            **phonebook:hull 20770**
-    Rena Hull - (301) 614-9654 - 155 Westway, Greenbelt, MD 20770

## Google Information - 2

**define:**
- definitions for the word or phrase that follows
- if definitions are available
- no space between **define:** and the word or phrase
-                      **define:nanotechnology**
**stocks:**
- Google treats the query terms as stock ticker symbols
- Links to a Yahoo finance page showing stock information for those symbols
- Ignores spaces

## Google Resources

- Free guides and FAQ
- Web searching in general
- Google’s features in specific
-     **http://www.google.com/help/**
- Google Usenet newsgroup
**    ****google.public.support.general**
***Google ******Hacks*** by Calishain and Dornfest
- Extremely advanced book written for Perl programmers
- Recommended but only if you can hack it

## Other Resources

- Deep Web (Invisible Web)
- Content of databases accessible on the Web
- Estimated 500 times larger than the fixed Web
- Non-textual files on the Web
- Multimedia files
- Graphical files (now offered by Google)
- Documents in non-standard formats such as Portable Document Format (PDF)
- Sources
- Directory of high quality databases maintained by Gary Price and Chris Sherman **http://www.invisible-web.net/**
- CompletePlanet  **http://www.completeplanet.com/**

## Other Resources - 2

- The GSFC Library     **http://library.gsfc.nasa.gov/**
- Library Catalog
- Article/Paper Search Engines
- Technical databases (deep web)
- Available to users within GSFC domains
- Journals, eBooks, and more
- NEW: Ask A Librarian
- Real-time reference!
- 10 am to 4 pm, Mon - Fri
- Never underestimate a well-trained reference librarian

## Future Goals

- Search engines understand (and answer) questions in natural language
- Microsoft Research’s AskMSR
- Rewrite search question (generate templates)
- Search for exact matches -- done
- Search for close matches -- redundancy => high probability
- Search engines as good as well-trained reference librarians
- Major advances needed in probabilistic machine learning and natural-language processing