# Presentation: 207435

## Find out what people have been doing on your web site

- Phil Dickerson, US EPA

## CSI Briefing Room

- Why investigate?
- The scene of the click: the evidence and what it tells you
- Wrapping up the Click Scene:  How you can “crack the case” in your outreach efforts
- What we don’t know
- Questions?

## Why Investigate?

- Without tracking web site usage, you’re leaving your excellent work “on the stump”
- Tracking will let you know things like:
- Which products are in highest demand
- What sorts of customers are visiting your web site
- How events affect usage, i.e. ozone or PM episodes

## Click Scene Terminology

*Hit*:  a SINGLE transaction between a web client, i.e. browser, and a web server
*Web log*:  text file that tracks every transaction between a client and server
*IP Address*:  the unique address of a single device on the Internet, similar to a telephone number
*URL*:  Uniform Resource Locator, more commonly known as “a web address”

## What We Know:  The Crime Scene

- Someone clicked into your site
- All they left behind was a few entries in a web log:
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /current/pm25/pm25_super_current_hour_small.gif HTTP/1.1" 200 25416 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /images/aqi_logo.gif HTTP/1.1" 200 2046 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /scripts/date.js HTTP/1.1" 200 1660 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /images/woodstove.gif HTTP/1.1" 200 9397 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /images/epa_seal_small.gif HTTP/1.1" 200 2130 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /current/super_current_hour_small.gif HTTP/1.1" 200 21374 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /current/current-outlook-small.gif HTTP/1.1" 200 37087 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /images/airnow_banner2.gif HTTP/1.1" 200 16582 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /images/exit.gif HTTP/1.1" 200 496 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET / HTTP/1.1" 200 40340 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:27 -0500] "GET /styles/epafiles_epastyles.css HTTP/1.1" 200 3303 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:28 -0500] "GET /images/noaa_logo.gif HTTP/1.1" 200 1960 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:28 -0500] "GET /images/nasa_logo.gif HTTP/1.1" 200 1614 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:28 -0500] "GET /images/nps_logo_small.gif HTTP/1.1" 200 1547 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:28 -0500] "GET /images/environ_canada.gif HTTP/1.1" 200 1030 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
- What does all that mean?

## Web Log Anatomy

- One line per transaction, or “hit”
- Each line has the following elements:
- IP Address of the requestor
- Date and time of the request
- The actual request
- Return code
- Number of bytes delivered
- Referring page
- Hardware/Software info

## Web Log Anatomy ... up close

- 	passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“
**passepartout.rtp.epa.gov:  **my computer’s “name”
**[12/Jan/2005:09:24:30 -0500]:  **The date and time, with GMT offset
**"GET /today/forecast.gif HTTP1.1”:  **What the browser or client requested
**404:  **The return code.  404 means Page Not Found!
**12183:  **12,183 bytes were returned.  In this case, they would comprise the EPA Page Not Found page
- _[http://airnow.gov/](http://airnow.gov/)__:  _The referring page
**Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“:  **The hardware and software used by the requestor

## What does each piece of evidence tell us about the click?

## passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“

- The requesting IP Address: passepartout.epa.gov
- This is the only evidence left at the scene that can be used to divine demographics
- In this example, we can readily see that the request came from EPA
- We can also sort the file on this address and follow what this user actually viewed on the site

## passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“

- The date and time:
**[12/Jan/2005:09:24:30 -0500]**

- Obviously, gives us the date and time
- Combined with other fields, can yield peak days of week, highest activity in a month or a year.
- Can also give us an idea of how long people “browse” pages

## passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“

- The Request:
**"GET /today/forecast.gif HTTP1.1”**

- Tells us exactly what file the browser requested
-  Gives us the version of the HTTP protocol used
- Enables the CSI to determine which files, graphics, or documents are most, and least, requested

## passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“

- The Return Code:
**“****404”**

-  Gives the result of the transaction – 404 means Page Not Found! This helps track old links to incorrect pages, or tells us when users attempt to type a URL from memory – incorrectly!
- Also, 404 has become techo-speak for a person who is utterly clueless, i.e. My supervisor is so 404.
-  Other codes:  200 = Successful  301 = Access Denied

- Number of Bytes:
**“****12183”**

-  The number of bytes sent in response to the request
- Can be used to compute total bytes transferred out of the web site

## passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“

- The Referring Page:
**"http://airnow.gov/"**

- The last page the browser requested, BEFORE the current request
- Tells the CSI what the “clicker” was doing just before entering our click scene.
- Can be used to determine what other sites link to your site
- Also tells you what “click path” a user followed

## passepartout.rtp.epa.gov - - [12/Jan/2005:09:24:30 -0500] "GET /today/forecast.gif HTTP/1.1" 404 12183 "http://airnow.gov/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“

- The Client Configuration:
**"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)“**

-  The browser type and version used by the requesting client
- The operating system the client was running
- Provides excellent “forensic” evidence as to the demographics of your users
- Can be used to determine which browsers to support
- Can even be used to see how many older operating systems are used by your audience

## OK, but I don’t like these logs as much as YOU do...

- How could you not?
- Well, if you really don’t want to look at logs
- Most IT groups do some log analysis using off-the-shelf software
- This software reads the log files and computes a variety of interesting metrics
- These web usage reports are usually done automatically and are readily available
- The reports make interesting and useful assumptions about web site visitors

## A Real-Life Sample from italladdsup.gov

**Timeframe:  **10/01/2004 00:00:00 - 10/31/2004 23:59:59
**Hits**
- Entire Site (Successful)  65,910
- Average Per Day  2,126
- Home Page  20
**Page Views**
- Page Views (Impressions)  26,146
- Average Per Day  843
- Document Views  855
**Visitor Sessions**
- Visitor Sessions  9,452
- Average Per Day  304
- Average Visitor Session Length  00:31:47
- International Visitor Sessions  3.44%
- Visitor Sessions of Unknown Origin  17.89%
- Visitor Sessions from United States  78.66%
**Visitors**
- Unique Visitors  2,834
- Visitors Who Visited Once  2,416
- Visitors Who Visited More Than Once  418

## HitsEntire Site (Successful)  65,910Average Per Day  2,126Home Page  20

- Hits encompass ANY request – including each and every graphic.
- Hits provide a broad, overall measure of web activity; however, simply adding a new logo or picture will increase hits, since each new graphic requires an extra transaction.
- “Home Page” is generally set within the software and may not refer to what you think is your home page.  Check with your IT department to see how this is set!

## Slide 18

## Slide 19

## Page ViewsPage Views (Impressions)  26,146Average Per Day  843Document Views  855

- Page views are a good measure of your site’s usage.  They are comprised only of requests for web pages, i.e. files that end in HTM or HTML.  Graphics are not counted
- Document views can be tricky. Most software requires that the IT folks tell it what a “document” is, by extension, i.e. .doc, .pdf, .txt, etc.
- Take home message:  Make sure your IT department has the proper file extensions marked as “documents” in their reporting software.

## Visitor SessionsVisitor Sessions  9,452Average Per Day  304Average Visitor Session Length  00:31:47International Visitor Sessions  3.44%Visitor Sessions of Unknown Origin  17.89%Visitor Sessions from United States  78.66%

- Visitor sessions are an artificial construct.  The web server sees only single requests, thus it does not know “users”.
- Log analyzers define a visitor session as a group of requests from the same client, with no more than a set time period between them – my analyzer uses 10 minutes.
- If a visitor clicks 1 page every 9 minutes, that is one visitor session.  If they click three pages, with 15 minutes between clicks, it would be three sessions.
- Session length is the elapsed time between their first request and their last request.  It gives you an idea of how long they looked around your site.
- Origin of user sessions is gleaned from the client computer’s address, i.e. .gov,.com,.edu = US, .ca = Canada, .jp = Japan, etc.

## VisitorsUnique Visitors  2,834Visitors Who Visited Once  2,416Visitors Who Visited More Than Once  418

- Visitors are really visiting browsers
- Each visitor is unique by their computer’s address, i.e. mybox.epa.gov
- “Unique” is limited to the scope of the analysis, so unique visitors may cover a month or a week, but is not tracked historically.
- Unique visitors gives a sense of how many *different *people are visiting your site.  This metric can help you determine if high activity periods are the result of a few very industrious users, or an influx of many different users.
- Visitors who visited once may indicate a random look, or someone who did not find what they needed
- Comparing single visits with multiple visits can tell you how many people come back on a regular basis

## The Top 5 Pieces of Evidence, IMHO

- IMHO is geek-speak for “In My Humble Opinion”
- Page views
- Document views and/or “top downloads”
- Unique visitors
- Referring sites or pages
- Activity by day (or hour, day of week)

## Other metrics you may see – and what they mean

- Most viewed pages
- Pages, i.e. HTML documents, and how many times they were requested, from most to least
- Referring Sites
- Each request lists the site just prior, so a list of sites that point to your site, and how many requests took that route, can be generated.
- Domain Report
- Requests, grouped by Internet domain.  Can assume “aol.com” is probably private citizen, or that any “.gov” is US government.  Also useful for measuring requests from other countries.

## Wrapping up your click scene

- Web statistics are essentially free market research
- Most government organizations are not allowed to survey the public, so web stats are valuable information
- When understood and used regularly, web stats can be an important component of your outreach “feedback loop” – telling you what worked and what didn’t and even offering hints as to why!

## What We DON’T Know...

- Web statistics are “stateless”, i.e. one hit has no connection to another.  Thus, we can only assume “visits” or “paths”
- The identity of a person – we only know their computer’s address
- More than ONE referring page prior to their visit to your site
- Anything about timing, other than the time an individual visit occurred – duration of visit is purely an assumption

## Questions?