Import GAE admin UI scrape utility

Summary:
gae_dashboard_curl.py fetches contents from the GAE admin UI, and
fetch_stats.sh configures it for use in a cron job and determines where
the output goes.

instance_report.py scrapes the HTML output for /instances and converts
it to JSON for later processing.

Test Plan:
Created ~/private_pw containing the account password that matches
$username in fetch_stats.sh and ran the script then verified that output
was written to the expected location.

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D798
Correct path to analytics source dir

Test Plan: Fingers crossed.

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D799
Install the Python library lxml

Summary:
The aws-config scripts for analytics rely on requirements.txt to
determine which Python libraries are installed.

lxml is needed for gae_dashboard/instance_report.py

GZIP stats JSON

It's text data that is much smaller gzipped.

Since the resulting file has the GZIP header, it won't be empty and we
can't trivially tell if it should be deleted. You will be able to which
files are invalid because they will be about 25 bytes on disk.

Test Plan:
Installed lxml manually on the analytics machine and ran
gae_dashboard/fetch_logs.sh which invokes the instance report.

Fingers crossed.

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D801
Support generation of multiple dashboard reports

Summary:
Refactor gae_dashboard_curl.py so that others can also create RPC
servers to GAE easily.

Refactor the scraping code out of instance_report.py into parsers.py so
that other reports can use it as well.

Create gae_dashboard_summary.py which will eventually be a single JSON
report containing data about the current state of all dashboard pages.

Test Plan:
I ran all the scripts to verify that they still worked, and that
instance_report.py generates the same output as before.

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D805
Add parser for /billing/history

Summary:
The parser isn't complete and only handles usage report events. The
data will be used to generate an analytics dashboard.

Test Plan:
I used gae_dashboard_curl.py to download /billing/history page then
tested manually in an interpreter with:
    import parsers, json, sys
    r = parsers.BillingHistory(open('tmp.html').read())
    json.dump(r.event_dicts(), sys.stdout)

Reviewers: benkomalo

Reviewed By: benkomalo

Differential Revision: http://phabricator.khanacademy.org/D849
Add script to import billing history usage data to mongo

Summary:
The shell script is meant to be run as a cron job each morning (PST) to
download daily usage reports from the GAE dashboard and load them into
mongo.

The python script reads an App Engine usage report in CSV from input
then detects which data isn't already in mongo db and if any is found,
loads it into mongo.

Test Plan:
I started a local mongod instance, changed the mongo IP to 'localhost'
in load_usage_reports.py, then ran ./fetch_usage.sh and verified that it
loaded the full history. Then I deleted the latest report from mongo and
re-ran to verify that only the latest report was imported.

Reviewers: benkomalo

Reviewed By: benkomalo

Differential Revision: http://phabricator.khanacademy.org/D857
Fix a typo of mongo DB name
Store App Engine instance stats in the database

Summary:
Before this, the instance reports are stored as compressed JSON on disk.
The problem is that loading these files and graphing them is an
expensive and slow operation.

By instead storing only the instance summaries in the analytics
database, rather than all instance data, the reporting UI can be much
faster, and we don't need all of the raw data anyway.

fetch_stats.py will now write to the database rather than disk, and the
corresponding dashboard has been modified to read from the database.

Furthermore, the App Engine dashboard parser for the instances page has
been updated to return parsed data for the ms, MBytes, and totals field
rather than the raw text. Now other tools don't have to duplicate this
logic.

Test Plan:
I modified instance_report.py to print the record rather than storing it
in the database, and then modified fetch_stats.py to use my credentials
and analytics source to download the summaries, and finally I ran
./fetch_stats.sh and verified that the expected result was output.

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D961
Download memcache statistics to analytics database

Summary:
fetch_stats.sh now downloads both instance and memcache stats. In the
future this should probably become a Python script.

memcache_report.py converts from the scraped HTML into a record of the
statistics and stores this in the analytics database.

parsers.py now supports parsing of the /memcache dashboard page.

Test Plan:
I set srcdir and username in fetch_stats.sh, modified instance_report.py
to print the record instead of writing to the db, and passed --debug to
memcache_report.py. Then I ran fetch_stats.sh and verified that the
expected information was printed.

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D971
Scrape data from GAE's main dashboard into mongo db

Summary:
parsers.py now accepts /dashboard markup and can extract the URLs
corresponding to the charts at the top of the page for 6, 12, and 24
hour charts.

The new dashboard_report.py can unpack the data in a Google Charts API
URL and turn it back into time series data for each of the charts.

Next up is aggregating the time series data and saving records that are
newer than what is in the database, then, fetch_stats.sh needs to be
updated to run the full pipeline.

Test Plan:
I ran:
<~/private_pw ./gae_dashboard_curl.py "https://appengine.google.com/dashboard?app_id=s~khan-academy" USERNAME >dashboard.html

to get a local copy of the dashboard, then tested output of:
<dashboard.html dashboard_report.py `date +%s`

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D998
Fix a bug around field labeling

As part of a previous review comment, I quickly snuck in a change to
strip chart label suffixes in a slick way.  However, the logic also
controlled which charts were returned and this caused an exception
to be thrown when unexpected chart labels (like
"Requests/Second (12 hrs)" rather than "Requests/Second") were returned.

I'm now sneaking in this back-out change.

Test plan:
I used a previously downloaded dashboard.html and ran this to verify
that the error was thrown before the fix went in and that the expected
results were printed after the fix:

<dashboard.html python dashboard_report.py --debug `date +%s`
Update comment about duplicate records

I'm giving up on de-duplicating records for now.  Having a few duplicate
data points doesn't throw off the dashboards terribly and really having
this data be accurate is difficult since we unpack it from a graph and
that is, by its nature, inaccurate.

I also found that while sometimes a duplicate record is added when the
script runs at 5-minute intervals, often the record is an
almost-duplicate where a few fields are different or missing.  I took a
few passes at this and had trouble fixing it.

Reviewers: None (just changing a comment)
Fix GAE SDK import in gae_dashboard_curl.py

Summary:
On the analytics machine, a PYPI module has been installed that provides
"google", but has no "appengine.tools" submodule. Other scripts had
already migrated to using gae_util.py's SDK lookup and were not
impacted, but this fetch_usage.sh (and other scripts depending on
gae_dashboard_curl.py) broke with the error:

Traceback (most recent call last):
  File "/home/analytics/analytics/gae_dashboard/gae_dashboard_curl.py", line 20, in <module>
    from google.appengine.tools import appengine_rpc
ImportError: No module named appengine.tools

I also updated the gae_dashboard/fetch_* scripts to be parameterizable,
it made local testing easier.

Test Plan:
$ cd analytics/gae_dashboard
$ srcdir=. ./fetch_usage.sh
Importing usage reports newer than 2014-04-01
Importing 0 documents

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D7951
Add --verbose and --dry-run to GAE scraping scripts

Summary:
These scrapers are failing. These options make it easier to debug
without worrying about writing to the database. They're a precursor to
fixing the scrapers.

I updated to argparse, and then brought the three up to roughly the same
place in terms of script structure and options.

Test Plan:
I ran each script individually with the new verbose and dry-run options.
They still fail, but this makes them easier to debug.

echo test | ./dashboard_report.py -nv 0
echo test | ./instance_report.py -nv 0
echo test | ./memcache_report.py -nv 0
srcdir=. ./fetch_stats.sh -nv

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D7970
Update instance_report.py scraper to match App Engine UI

Summary:
This fixes 3 issues:
1) There is a new field for the App Engine version.
2) During new version rollout there are 2 rows of instance summaries.
3) Instance summary data is no longer present on /instances, it is
   delay-loaded by the web UI by calling /instance_summary.

Test Plan:
I downloaded the instance summary HTML and ran it against the script in
"dry-run" and "verbose" mode:

  $ ./gae_dashboard_curl.py 'https://appengine.google.com/instance_summary?app_id=s~khan-academy' chris@khanacademy.org >instance_summary.html
  $ <instance_summary.html ./instance_report.py -vn 0

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D8018
Update memcache_report.py to match App Engine UI

Summary:
This required 3 fixes:
1) Some extraction required handling <td>Some <span>text</span></td>
   rather than just <td>Some text</td>.
2) The "oldest item age" format is no longer only "second(s)".
3) Total cache size no longer lists "byte(s)".

Test Plan:
  $ ./gae_dashboard_curl.py 'https://appengine.google.com/memcache?app_id=s~khan-academy' chris@khanacademy.org >memcache.html
  $ <memcache.html ./memcache_report.py -vn 0

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D8024
Update dashboard_report.py scraper to current App Engine UI

Summary:
There were 2 issues to fix:
1) Many reports changed names, although most values remained the same.
2) The default graph rendered on the page is now for the '24 hrs' period.

This isn't back to par, since it used to collect far more
data. Currently, only the "Summary" graph URL is contained in the
page. Before, all graphs for "6 hrs" were in the page. There appears to
be a new "/stats" endpoint that generates the graph URLs. In the future
we should hit that directly.

Test Plan:
  $ ./gae_dashboard_curl.py 'https://appengine.google.com/dashboard?app_id=s~khan-academy' chris@khanacademy.org >dashboard.html
  $ <dashboard.html ./dashboard_report.py -vn 0

Reviewers: csilvers

Reviewed By: csilvers

Differential Revision: http://phabricator.khanacademy.org/D8029
Send data to graphite as well as to mongo-db.

Summary:
Eventually, we can move away from mongo entirely, if we end up using
graphite for everything.  Happy days that would be!

We use the pickle protocol (over TCP) for maximum efficiency.

Test Plan:
Ran this script on the analytics machine, and saw the new key-names
show up at
   https://www.hostedgraphite.com/accounts/metrics/delete/preview/?pattern=*

Reviewers: chris

Reviewed By: chris

Differential Revision: http://phabricator.khanacademy.org/D8362
Add send-to-graphite support to *all* the gae-scraping scripts!

Summary:
We now have a utility file to do this work since three different
scripts need to do it.

Test Plan:
Set up the various secrets in my homedir and ran
   env srcdir=. username=prod-deploy@khanacademy.org ./fetch_stats.sh -v
and saw it complete successfully.

Reviewers: chris

Reviewed By: chris

Differential Revision: http://phabricator.khanacademy.org/D8364
Shorten the top-of-file docstring, and put the global vars earlier.

Summary:
Fix the dashboard scraper now that the dashboard has lots of urls.

Now instead of sending in an html page for the python script to parse,
we actually fetch the chart-data directly in the shell script, for
each of the 14 charts that GAE shows on the admin consol 'dashboards'
page.

We then bundle all that chart-data up into a giant json struct and
send *that* to the python script.  It's kinda fun, in a scray kind of
way, to construct json via shell 'echo' commands. :-}

Since we're passing in json, we now have an opportunity to pass in
some other info as well, that we used to just have to manually keep in
sync between the shell script and the python script, such as how big a
time-window the chart should be (6 hrs, 24 hrs, etc).

Make _time_windows just a list.

We never index it by name, so no need for it to be an ordered-dict.

Test Plan:
Set up all the necessary secrets on my home machine and ran
   env srcdir=. username=prod-deploy@khanacademy.org ./fetch_stats.sh -v

and saw it emit this:

  {'average_memory_mb': 297.1, 'average_qps': 0.504, 'num_instances': 747, 'utc_datetime': datetime.datetime(2014, 4, 18, 22, 49, 43), 'average_latency_ms': 404.1}
  {'oldest_item_age_seconds': 103879, 'utc_datetime': datetime.datetime(2014, 4, 18, 22, 49, 53), 'miss_count': 12835081, 'hit_ratio': 0.87, 'item_count': 123295, 'hit_count': 86659639, 'total_cache_size_bytes': 4288114546}
  [{'milliseconds_used_per_second': 238700.0, 'memcache_ops_per_second': 3410.0, 'dynamic_requests_per_second': 410.0, 'memcache_total_bytes_per_second': 52290000.0, 'milliseconds_per_dynamic_request': 582.4, 'api_cpu_seconds_used_per_second': 0.09768, 'active_instance_count': 223.1, 'utc_datetime': datetime.datetime(2014, 4, 18, 22, 27, 1), 'memcache_compute_units_per_second': 8630.0, 'billed_instance_count': 920.5, 'memcache_bytes_sent_per_second': 13550000.0, 'server_errors_per_second': 0.1062, 'milliseconds_per_loading_request': 9604.0, 'total_cpu_seconds_used_per_second': 182.0, 'dos_api_denials_per_second': 0.0, 'bytes_sent_per_second': 10800000.0, 'cached_requests_per_second': 283.8, 'static_requests_per_second': 8.303, 'client_errors_per_second': 3.2, 'memcache_size_mb': 4087.0, 'memory_usage_mb': 251300.0, 'memcache_bytes_received_per_second': 38740000.0, 'total_instance_count': 1436.0, 'bytes_received_per_second': 1211000.0, 'quota_denials_per_second': 0.0}, {'milliseconds_used_per_second': 196400.0, 'memcache_ops_per_second': 3392.0, 'dynamic_requests_per_second': 409.5, 'memcache_total_bytes_per_second': 49050000.0, 'milliseconds_per_dynamic_request': 479.9, 'api_cpu_seconds_used_per_second': 0.09768, 'active_instance_count': 179.1, 'utc_datetime': datetime.datetime(2014, 4, 18, 22, 31, 56), 'memcache_compute_units_per_second': 8488.0, 'billed_instance_count': 890.5, 'memcache_bytes_sent_per_second': 13580000.0, 'server_errors_per_second': 0.1062, 'milliseconds_per_loading_request': 5007.0, 'total_cpu_seconds_used_per_second': 180.4, 'dos_api_denials_per_second': 0.0, 'bytes_sent_per_second': 9978000.0, 'cached_requests_per_second': 271.3, 'static_requests_per_second': 5.372, 'client_errors_per_second': 2.821, 'memcache_size_mb': 4088.0, 'memory_usage_mb': 266100.0, 'memcache_bytes_received_per_second': 35440000.0, 'total_instance_count': 1429.0, 'bytes_received_per_second': 1192000.0, 'quota_denials_per_second': 0.0}, {'milliseconds_used_per_second': 193800.0, 'memcache_ops_per_second': 3419.0, 'dynamic_requests_per_second': 399.5, 'memcache_total_bytes_per_second': 52320000.0, 'milliseconds_per_dynamic_request': 485.3, 'api_cpu_seconds_used_per_second': 0.09768, 'active_instance_count': 183.9, 'utc_datetime': datetime.datetime(2014, 4, 18, 22, 36, 52), 'memcache_compute_units_per_second': 8615.0, 'billed_instance_count': 900.7, 'memcache_bytes_sent_per_second': 13610000.0, 'server_errors_per_second': 0.08608, 'milliseconds_per_loading_request': 6429.0, 'total_cpu_seconds_used_per_second': 173.3, 'dos_api_denials_per_second': 0.0, 'bytes_sent_per_second': 9944000.0, 'cached_requests_per_second': 279.6, 'static_requests_per_second': 5.617, 'client_errors_per_second': 4.61, 'memcache_size_mb': 4087.0, 'memory_usage_mb': 222000.0, 'memcache_bytes_received_per_second': 38680000.0, 'total_instance_count': 1263.0, 'bytes_received_per_second': 1182000.0, 'quota_denials_per_second': 0.0}]
  Importing 3 records

which seems plausible to me.  I'm not sure why it was getting 15
minutes' worth of data rather than just 5, but kinda fun to see that
part of the code is still working.

None

(same)

Reviewers: chris

Reviewed By: chris

Differential Revision: http://phabricator.khanacademy.org/D8368