CVE-2021-4276: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection') (CWE-89)
value: ** UNSUPPORTED WHEN ASSIGNED ** A vulnerability was found in dns-stats hedgehog. It has been rated as problematic. Affected by this issue is the function DSCIOManager::dsc_import_input_from_source of the file src/DSCIOManager.cpp. The manipulation leads to sql injection. The attack may be launched remotely. The exploit has been disclosed to the public and may be used. The real existence of this vulnerability is still doubted at the moment. The name of the patch is 58922c345d3d1fe89bb2020111873a3e07ca93ac. It is recommended to apply a patch to fix this issue. VDB-216746 is the identifier assigned to this vulnerability. NOTE: This vulnerability only affects products that are no longer supported by the maintainer. NOTE: We do assume that the Data Manager server can only be accessed by authorised users. Because of this, we don’t believe this specific attack is possible without such a compromise of the Data Manager server.
CWE description: The product constructs all or part of an SQL command using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended SQL command when it is sent to a downstream component. Without sufficient removal or quoting of SQL syntax in user-controllable inputs, the generated SQL query can cause those inputs to be interpreted as SQL instead of ordinary user data.
CVSS v3 base score: 8.8
Published: 2022-12-25T11:15Z
Repository: https://github.com/dns-stats/hedgehog
Language: Other
Files changed: .gitignore, CHANGELOG, KNOWN_ISSUES, LICENSE, Makefile.am, README, Vagrantfile, autogen.sh, conf/Makefile.am, conf/hedgehog.yaml.in, conf/nodes.csv, conf/service_addr.csv, conf/tlds.d/cctlds.csv, conf/tlds.d/idn_cctlds.csv, conf/tlds.d/legacy_gtlds.csv, conf/tlds.d/new_gtlds-20131023-20150611.csv, conf/tlds.d/new_gtlds-20150614-20151223.csv, configure.ac, docs/Hedgehog_Upgrade_Guide_(to_Packages).pdf, docs/src/Hedgehog_Installation_Guide_(from_Packages).xml, docs/src/Hedgehog_Tools_Guide.xml, docs/src/Hedgehog_Upgrade_Guide_(to_Packges).xml, docs/src/Hedgehog_User_Guide.xml, m4/acx_components.m4, m4/acx_libpostgres.m4, m4/acx_libpqxx.m4, m4/boost.m4, provision/compile.sh, provision/dependencies.sh, provision/launch.sh, src/DSCDataManager.cpp, src/DSCDataManager.h, src/DSCIOManager.cpp, src/DSCIOManager.h, src/DSCStrategy.cpp, src/DSCStrategy.h, src/DSCStrategyFactory.cpp, src/DSCStrategyFactory.h, src/Format1DSCStrategy.cpp, src/Format1DSCStrategy.h, src/Format2DSCStrategy.cpp, src/Format2DSCStrategy.h, src/Format3aDSCStrategy.cpp, src/Format3aDSCStrategy.h, src/Format3bDSCStrategy.cpp, src/Format3bDSCStrategy.h, src/Format4DSCStrategy.cpp, src/Format4DSCStrategy.h, src/Makefile.am, src/dsc-extractor.cpp, src/dsc_types.h, src/rssac_server/rssac.py, src/rssac_server/rssac_server.in, src/rssac_server/rssacd.conf, src/rssac_server/rssacd.in, src/rssac_server/rssacd.upstart.in, src/rssac_server/threadpool.py, tools/Makefile.am, tools/hedgehog/DDL_updates/000010_ddl_python.in, tools/hedgehog/DDL_updates/000011_ddl_new_graphs.in, tools/hedgehog/DDL_updates/000012_ddl_lower_key_index.in, tools/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in, tools/hedgehog/DDL_updates/000014_ddl_service_addr.in, tools/hedgehog/DDL_updates/000015_ddl_byregion.in, tools/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in, tools/hedgehog/DDL_updates/000017_ddl_node_stats.in, tools/hedgehog/DDL_updates/README.txt, tools/hedgehog/check_user_perms.in, tools/hedgehog/conf_read.in, tools/hedgehog/database_init.in, tools/hedgehog/database_manage_partitions.in, tools/hedgehog/database_process_rssac_data.in, tools/hedgehog/database_rm_old_rssac_data.in, tools/hedgehog/database_update_geoip.in, tools/hedgehog/database_update_nodes.in, tools/hedgehog/database_update_service_addr.in, tools/hedgehog/database_update_tlds.in, tools/hedgehog/database_update_tlds_from_zone.in, tools/hedgehog/datafiles_create_summary.in, tools/hedgehog/datafiles_rm_empty_xml_dirs.in, tools/hedgehog/datafiles_tar_old_xml_files.in, tools/hedgehog/hedgehog_database_create, tools/hedgehog/plotcache_generate_cached_plots.in, tools/hedgehog/plotcache_rm_cached_plots.in, tools/hedgehog/rssac_generate_reports.in, tools/hedgehog/sql/dns.sql, tools/hedgehog/sql/initialise_database.sql, tools/hedgehog/sql/insert_lookup_data.sql, tools/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in, tools/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in, tools/hedgehogctl.in, tools/refile_and_grok.in, www/Makefile.am, www/hedgehog/R_packages/RPostgreSQLHelper/DESCRIPTION, www/hedgehog/R_packages/RPostgreSQLHelper/NAMESPACE, www/hedgehog/R_packages/RPostgreSQLHelper/R/dbGetFunctions.r, www/hedgehog/R_packages/RPostgreSQLHelper/man/RPostgreSQLHelper-package.Rd, www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetDataFrame.Rd, www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetResultSet.Rd, www/hedgehog/R_packages/RPostgreSQLHelper/man/maxAttempts.Rd, www/hedgehog/conf/hedgehog.conf.in, www/hedgehog/conf/hedgehog_gui.yaml, www/hedgehog/conf/hedgehog_webdav.conf, www/hedgehog/conf/rssac.yaml, www/hedgehog/www/R/generatePlot.R, www/hedgehog/www/R/prepareStatement.R, www/hedgehog/www/R/startup.R.in, www/hedgehog/www/R/utilities.R, www/hedgehog/www/about.html.in, www/hedgehog/www/brew/hedgehog.brew.in, www/hedgehog/www/hedgehog.html.in, www/hedgehog/www/js/datetimes.js, www/hedgehog/www/js/gengraph.js, www/hedgehog/www/js/nodetabs.js, www/hedgehog/www/stylesheets/base.css, www/hedgehog/www/stylesheets/layout.css, www/hedgehog/www/stylesheets/skeleton.css
Commit date: 2021-05-13 16:49:28 +0100
Commit message: Merge pull request #190 from japroc/fix-potential-sqli-through-fs-path

Fix potential sql injection through fs path
Diff: diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..f13f951
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,21 @@
+#Vagrant
+.vagrant
+._*
+
+# Generated automatically
+/build
+Makefile.in
+/*.cache
+/aclocal.m4
+/compile
+/config.guess
+/config.h.*
+/config.sub
+/configure
+/depcomp
+/install-sh
+/ltmain.sh
+/missing
+/m4/libtool.m4
+/m4/lt*
+/src/rssac_server/*.pyc
diff --git a/CHANGELOG b/CHANGELOG
new file mode 100755
index 0000000..09d031f
--- /dev/null
+++ b/CHANGELOG
@@ -0,0 +1,203 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+
+Hedgehog 2.4.2 - 2019-01-08
+----------------------------
+* Improve validation of URL parameters to fix a SQL injection attack. The attack
+  can potentially leak data available to the user that the web server connects
+  to the database as. In the default configuration this is the 'www-data'
+  read-only user with access to only the Hedgehog database.
+
+
+Hedgehog 2.4.1 - 2018-05-08
+----------------------------
+* The only change is an update to the packaging that switches the postgres 9.3
+  packages from 'depends' to 'suggests' to allow the user to choose the DB 
+  version.
+
+
+Hedgehog 2.4.0 - 2017-08-21
+----------------------------
+* Improved error handling when trying to insert invalid data into db 
+* Renamed Node Statistics to Query Statistics in plot dropdown
+* New configuration option to hide 'node' selection on web interface
+* Add new graph for Server IP address vs transport (UDP/TCP)
+* Changed license to MPL
+* Added ability to re-name and remove nodes
+* Tested with Postgesql-9.6
+
+
+Hedgehog 2.3.0 - 2016-09-12
+----------------------------
+* Add ability to filter the server addresses displayed in the 'By server IP
+  address' plot. Controlled by the database_update_service_addr script.
+* Add new 'By region' graph which is now the default. This graph always groups
+  the results by Region. The 'By node' graph now always displays a line per node
+  regardless of the node selection (previously it grouped by Region when all
+  nodes were selected).
+* Default plot is now a configuration option. To specify a plot use the 'Hedgehog
+  internal plot name' as listed in the User Guide.
+* Update RSSAC report header format to match RSSC002v3 report.
+* Enhancements to datafiles_create_summary script.
+
+
+Hedgehog 2.2.0 - 2016-08-31
+-----------------------------
+* Added new dataset and plot dnskey_vs_asn
+
+
+Hedgehog 2.1.1 - 2016-08-16
+-----------------------------
+* Fix issue with database_manage_partitions script where generating tables for
+  -09 and -10 months didn't work
+* Add additional check on input parameters
+* Modify instructions due to issue with latest build of rcpp
+* Migrate to install from packages only from 2.1.1
+
+
+Hedgehog 2.1.0 - 2016-06-20
+------------------------------
+* Production release of 'Daurian'
+
+
+Hedgehog 2.1.0rc3 - 2016-06-01
+------------------------------
+* Add option to specify number of processes for refile_and_grok to use
+* Fix connection issue in update_tlds_from_zone script
+* Make hedgehogctl produce correct tools list for split installs
+* Minor documentation fixes
+
+
+Hedgehog 2.1.0rc2 - 2016-05-19
+------------------------------
+* Add default server for drop down as a config option
+* Fix alphabetical sorting of Country/City/Instance in node tab
+* Fix detection of correct user when running from cron jobs
+
+
+Hedgehog 2.1.0rc1 - 2016-05-09
+------------------------------
+* Updated packaging now that RPostgreSQL is available (thanks to Ondrej Sury)
+
+
+Hedgehog 2.1.0b2 - 2016-04-20
+-----------------------------
+
+* Add release name of "Daurian" for 2.1
+* Add script to update TLDs from IANA
+* Updates to documentation
+* Bug fix to handling of nodes.csv input
+* Ensure output format consistent with latest RSSAC02 specs
+* Fix issues with rssacd
+* Support '.' in server names
+* Move webdav config
+* Fix issues with jquery dependancies
+
+
+Hedgehog 2.1.0b1 - 2016-02-01 
+------------------------------
+
+Features:
+* Added new graphs for different types of TLD (QTYPE and RCODE for most
+  popular TLD). Some support for displaying IDN in their unicode glyphs.
+* Added new graph to display Geo-locations of querying subnets and nodes
+* Added new graphs to display query sources as BGP prefixes or ASN
+* Added new graphs to group nodes by Country/City/Instance
+* Added new graph to display Destination Address vs Query rate
+* Added option to group nodes in selection tab by Country/City/Instance and 
+  config option for default choice (specified in nodes.csv).
+* Added rssacd demon to collect rssac load-time and zone-size data. Requires
+  management IP Address of node to be specified in nodes.csv
+* Added GeoLite2 data created by MaxMind, available from
+  http://www.maxmind.com
+* Added static link URI to provide access to raw plot files
+* Updated database_update_nodes script to support new fields. NOTE: New format
+  is not backwards compatible!
+* Multiple remote Web front ends now supported
+* New config option to customise 'Support' link on Web page
+
+Maintenance/Bug fixes:
+* Improve DB performance by adding further indexes
+* Change to rssac scripts. Processing of unique_sources data now done by
+  'database_process_rssac_data' script which must be explicitly called.
+* Wrote check_user_perms to do all user checks, check db access and data dir.
+  access. Used in new database_update_nodes will gradually be used by all tools.
+* Improved chart sizes, font sizes and data ordering in some graphs
+* Significant enhancements to installation and management scripts and 
+  configuration files.
+* Fixed appearance of gvis column chart vAxis
+* Fixed error in rssac reports where ".root-servers.org" was used instead of 
+ ".root-servers.net". Also TCP length field is no longer included in the size
+  of the zone. 
+
+
+Hedgehog 2.0.0rc2 - 2015-07-09
+------------------------------
+
+* Add documentation source for packaging.
+
+Hedgehog 2.0.0rc1 - 2015-05-29
+------------------------------
+
+* Allow building of just the importer or just the web interface by adding
+  --disable-web and --disable-data-manager options to configure
+* Installed tools in libexec/hedgehog/ and removed hedgehog prefix and 
+  .sh extension
+* Created a hedgehogctl control script to wrap the tools.
+* Renamed several tools and made arguments consistent. Separate the RSSAC report
+  scripts to better align with split installs
+* Improved install instructions
+* Removed the install-rpg target and made installation of the RPostgreSQLHelper
+  consistent with other R packages that need installing.
+* Add missing check for SED in configure
+
+Hedgehog 2.0.0b2 - 2015-05-05
+-----------------------------
+
+* Support for Vagrant Provisioning (thanks melalj)
+* Various installation changes to support packaging (thanks arhbkb)
+* Changed the yaml parsing to use the Perl YAML::Tiny library
+* Changed the default_interactive_plot_type to svg (instead of flash)
+* Static Traffic Volume plot is now faceted by Transport vs IP version
+* Traffic Query/Response Difference plots added (requires R package dplyr)
+* RPostgreSQLHelper installed as separate package. This requires running
+ 'make install ; make install-rpg' on installation now.
+* Added vertical axis labels to googlevis line charts
+* Added node_id index to data tables
+* Added support for connecting to a remote database and password support
+* Fixed color palette in annotated charts
+* Work around for bug: mages/googleVis#14
+* Improved old xml tar script
+* Improvements to database creation and upgrade scripts
+
+Hedgehog 2.0.0b1 - 2014-09-09
+-----------------------------
+
+* Database schema changes: Plot table split into 3 and replaced by a view.
+  Added explicit handling of plots based on 2 datasets.
+  ****** Database version bumped - not backwards compatible *****
+* Added new config option 'default_interactive_plot_type' to 
+  hedgehog_gui.yaml. This can be used to generate SVG interactive plots
+  instead of the default interactive plots which require Flash.
+* Issue-#1: hedgehog_gui.yaml configuration file is now installed in
+  <prefix>/etc/hedgeghog
+* Database version mismatch now reported correctly in the GUI
+
+
+Hedgehog 2.0.0a1 - 2014-08-27
+----------------------------
+
+* Initial open source alpha release including:
+  * Data importer (refile_and_grok.sh)
+    - DAT-> Postgres database importer
+    - XML-> Postgres database importer
+    - XML-> DAT converter
+  * Database schema and scripts
+  * Web GUI
+  * Support for multiple servers
+  * Support for import of RSSAC data (from XML only)
diff --git a/KNOWN_ISSUES b/KNOWN_ISSUES
new file mode 100755
index 0000000..3a19d09
--- /dev/null
+++ b/KNOWN_ISSUES
@@ -0,0 +1,44 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+
+KNOWN ISSUES/LIMITATIONS:
+-------------------------
+
+* The full list of possible plots are displayed via the drop down menu in the
+  GUI, regardless of whether there is data in the database for the plot
+
+* A configuration option is available (default_interactive_plot_type in 
+  hedgehog_gui.yaml) which can specify the system to produce SVG or 
+  Flash plots for the interactive plots (google charts). There is a known 
+  issue with the SVG plots where long legends are not wrapped properly:
+  https://code.google.com/p/google-visualization-api-issues/issues/detail?id=1536
+  This affects, for example, the 'by_node' plot. However, we prefer SVG
+  to flash given all the flash vulnerabilities of late. 
+
+* Support for RSSAC reports is added for XML imports only, however extended 
+  RCODES cannot be collected by the existing DSC collector so only base RCODES
+  are collected and displayed. If the DSC collector is extended then further
+  work must be done in Hedgehog to also support extended RCODES
+
+* The Unique Sources plot is not available until after the RSSAC report has
+  been generated for the date in question.
+
+* If a user touches any controls while a query is waiting to return
+  then the 'generate graph' button becomes enabled again and a second
+  plot can be generated that will then over write the first.
+
+* No static plots are produced for the GEO type plots.The geochart is ignoring 
+  the instruction in the html not to show the color code in the tooltips.
+
+* Server, node and group and instance names may only contain alphanumeric 
+  characters, full stops (.) and hyphens (-). Country and city names can
+  only contain alphanumeric characters, spaces and hyphens (-)
+  
+* The Busiest client locations by city plot does't support passing the required
+  API key to the script element - see
+  https://developers.google.com/maps/documentation/javascript/error-messages#missing-key-map-error
diff --git a/LICENSE b/LICENSE
new file mode 100755
index 0000000..a41a5f5
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,445 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+This project contains code from the following projects:
+
+Boost
+# Boost Software License - Version 1.0 - August 17th, 2003
+# 
+# Permission is hereby granted, free of charge, to any person or organization
+# obtaining a copy of the software and accompanying documentation covered by
+# this license (the "Software") to use, reproduce, display, distribute,
+# execute, and transmit the Software, and to prepare derivative works of the
+# Software, and to permit third-parties to whom the Software is furnished to
+# do so, all subject to the following:
+# 
+# The copyright notices in the Software and this entire statement, including
+# the above license grant, this restriction and the following disclaimer,
+# must be included in all copies of the Software, in whole or in part, and
+# all derivative works of the Software, unless such copies or derivative
+# works are solely in the form of machine-executable object code generated by
+# a source language processor.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
+# SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
+# FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+
+
+boost.m4
+# boost.m4: Locate Boost headers and libraries for autoconf-based projects.
+# Copyright (C) 2007, 2008, 2009, 2010, 2011  Benoit Sigoure <tsuna@lrde.epita.fr>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Additional permission under section 7 of the GNU General Public
+# License, version 3 ("GPLv3"):
+#
+# If you convey this file as part of a work that contains a
+# configuration script generated by Autoconf, you may do so under
+# terms of your choice.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Skeleton CSS
+/*
+* Skeleton V1.2
+* Copyright 2011, Dave Gamache
+* www.getskeleton.com
+* Free to use under the MIT license.
+* http://www.opensource.org/licenses/mit-license.php
+* 6/20/2012
+*/
+
+############################################################################
+
+Mozilla Public License Version 2.0
+==================================
+
+1. Definitions
+--------------
+
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
+
+1.2. "Contributor Version"
+    means the combination of the Contributions of others (if any) used
+    by a Contributor and that particular Contributor's Contribution.
+
+1.3. "Contribution"
+    means Covered Software of a particular Contributor.
+
+1.4. "Covered Software"
+    means Source Code Form to which the initial Contributor has attached
+    the notice in Exhibit A, the Executable Form of such Source Code
+    Form, and Modifications of such Source Code Form, in each case
+    including portions thereof.
+
+1.5. "Incompatible With Secondary Licenses"
+    means
+
+    (a) that the initial Contributor has attached the notice described
+        in Exhibit B to the Covered Software; or
+
+    (b) that the Covered Software was made available under the terms of
+        version 1.1 or earlier of the License, but not also under the
+        terms of a Secondary License.
+
+1.6. "Executable Form"
+    means any form of the work other than Source Code Form.
+
+1.7. "Larger Work"
+    means a work that combines Covered Software with other material, in 
+    a separate file or files, that is not Covered Software.
+
+1.8. "License"
+    means this document.
+
+1.9. "Licensable"
+    means having the right to grant, to the maximum extent possible,
+    whether at the time of the initial grant or subsequently, any and
+    all of the rights conveyed by this License.
+
+1.10. "Modifications"
+    means any of the following:
+
+    (a) any file in Source Code Form that results from an addition to,
+        deletion from, or modification of the contents of Covered
+        Software; or
+
+    (b) any new file in Source Code Form that contains any Covered
+        Software.
+
+1.11. "Patent Claims" of a Contributor
+    means any patent claim(s), including without limitation, method,
+    process, and apparatus claims, in any patent Licensable by such
+    Contributor that would be infringed, but for the grant of the
+    License, by the making, using, selling, offering for sale, having
+    made, import, or transfer of either its Contributions or its
+    Contributor Version.
+
+1.12. "Secondary License"
+    means either the GNU General Public License, Version 2.0, the GNU
+    Lesser General Public License, Version 2.1, the GNU Affero General
+    Public License, Version 3.0, or any later versions of those
+    licenses.
+
+1.13. "Source Code Form"
+    means the form of the work preferred for making modifications.
+
+1.14. "You" (or "Your")
+    means an individual or a legal entity exercising rights under this
+    License. For legal entities, "You" includes any entity that
+    controls, is controlled by, or is under common control with You. For
+    purposes of this definition, "control" means (a) the power, direct
+    or indirect, to cause the direction or management of such entity,
+    whether by contract or otherwise, or (b) ownership of more than
+    fifty percent (50%) of the outstanding shares or beneficial
+    ownership of such entity.
+
+2. License Grants and Conditions
+--------------------------------
+
+2.1. Grants
+
+Each Contributor hereby grants You a world-wide, royalty-free,
+non-exclusive license:
+
+(a) under intellectual property rights (other than patent or trademark)
+    Licensable by such Contributor to use, reproduce, make available,
+    modify, display, perform, distribute, and otherwise exploit its
+    Contributions, either on an unmodified basis, with Modifications, or
+    as part of a Larger Work; and
+
+(b) under Patent Claims of such Contributor to make, use, sell, offer
+    for sale, have made, import, and otherwise transfer either its
+    Contributions or its Contributor Version.
+
+2.2. Effective Date
+
+The licenses granted in Section 2.1 with respect to any Contribution
+become effective for each Contribution on the date the Contributor first
+distributes such Contribution.
+
+2.3. Limitations on Grant Scope
+
+The licenses granted in this Section 2 are the only rights granted under
+this License. No additional rights or licenses will be implied from the
+distribution or licensing of Covered Software under this License.
+Notwithstanding Section 2.1(b) above, no patent license is granted by a
+Contributor:
+
+(a) for any code that a Contributor has removed from Covered Software;
+    or
+
+(b) for infringements caused by: (i) Your and any other third party's
+    modifications of Covered Software, or (ii) the combination of its
+    Contributions with other software (except as part of its Contributor
+    Version); or
+
+(c) under Patent Claims infringed by Covered Software in the absence of
+    its Contributions.
+
+This License does not grant any rights in the trademarks, service marks,
+or logos of any Contributor (except as may be necessary to comply with
+the notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+No Contributor makes additional grants as a result of Your choice to
+distribute the Covered Software under a subsequent version of this
+License (see Section 10.2) or under the terms of a Secondary License (if
+permitted under the terms of Section 3.3).
+
+2.5. Representation
+
+Each Contributor represents that the Contributor believes its
+Contributions are its original creation(s) or it has sufficient rights
+to grant the rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+This License is not intended to limit any rights You have under
+applicable copyright doctrines of fair use, fair dealing, or other
+equivalents.
+
+2.7. Conditions
+
+Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
+in Section 2.1.
+
+3. Responsibilities
+-------------------
+
+3.1. Distribution of Source Form
+
+All distribution of Covered Software in Source Code Form, including any
+Modifications that You create or to which You contribute, must be under
+the terms of this License. You must inform recipients that the Source
+Code Form of the Covered Software is governed by the terms of this
+License, and how they can obtain a copy of this License. You may not
+attempt to alter or restrict the recipients' rights in the Source Code
+Form.
+
+3.2. Distribution of Executable Form
+
+If You distribute Covered Software in Executable Form then:
+
+(a) such Covered Software must also be made available in Source Code
+    Form, as described in Section 3.1, and You must inform recipients of
+    the Executable Form how they can obtain a copy of such Source Code
+    Form by reasonable means in a timely manner, at a charge no more
+    than the cost of distribution to the recipient; and
+
+(b) You may distribute such Executable Form under the terms of this
+    License, or sublicense it under different terms, provided that the
+    license for the Executable Form does not attempt to limit or alter
+    the recipients' rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+You may create and distribute a Larger Work under terms of Your choice,
+provided that You also comply with the requirements of this License for
+the Covered Software. If the Larger Work is a combination of Covered
+Software with a work governed by one or more Secondary Licenses, and the
+Covered Software is not Incompatible With Secondary Licenses, this
+License permits You to additionally distribute such Covered Software
+under the terms of such Secondary License(s), so that the recipient of
+the Larger Work may, at their option, further distribute the Covered
+Software under the terms of either this License or such Secondary
+License(s).
+
+3.4. Notices
+
+You may not remove or alter the substance of any license notices
+(including copyright notices, patent notices, disclaimers of warranty,
+or limitations of liability) contained within the Source Code Form of
+the Covered Software, except that You may alter any license notices to
+the extent required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+You may choose to offer, and to charge a fee for, warranty, support,
+indemnity or liability obligations to one or more recipients of Covered
+Software. However, You may do so only on Your own behalf, and not on
+behalf of any Contributor. You must make it absolutely clear that any
+such warranty, support, indemnity, or liability obligation is offered by
+You alone, and You hereby agree to indemnify every Contributor for any
+liability incurred by such Contributor as a result of warranty, support,
+indemnity or liability terms You offer. You may include additional
+disclaimers of warranty and limitations of liability specific to any
+jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+---------------------------------------------------
+
+If it is impossible for You to comply with any of the terms of this
+License with respect to some or all of the Covered Software due to
+statute, judicial order, or regulation then You must: (a) comply with
+the terms of this License to the maximum extent possible; and (b)
+describe the limitations and the code they affect. Such description must
+be placed in a text file included with all distributions of the Covered
+Software under this License. Except to the extent prohibited by statute
+or regulation, such description must be sufficiently detailed for a
+recipient of ordinary skill to be able to understand it.
+
+5. Termination
+--------------
+
+5.1. The rights granted under this License will terminate automatically
+if You fail to comply with any of its terms. However, if You become
+compliant, then the rights granted under this License from a particular
+Contributor are reinstated (a) provisionally, unless and until such
+Contributor explicitly and finally terminates Your grants, and (b) on an
+ongoing basis, if such Contributor fails to notify You of the
+non-compliance by some reasonable means prior to 60 days after You have
+come back into compliance. Moreover, Your grants from a particular
+Contributor are reinstated on an ongoing basis if such Contributor
+notifies You of the non-compliance by some reasonable means, this is the
+first time You have received notice of non-compliance with this License
+from such Contributor, and You become compliant prior to 30 days after
+Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+infringement claim (excluding declaratory judgment actions,
+counter-claims, and cross-claims) alleging that a Contributor Version
+directly or indirectly infringes any patent, then the rights granted to
+You by any and all Contributors for the Covered Software under Section
+2.1 of this License shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all
+end user license agreements (excluding distributors and resellers) which
+have been validly granted by You or Your distributors under this License
+prior to termination shall survive termination.
+
+************************************************************************
+*                                                                      *
+*  6. Disclaimer of Warranty                                           *
+*  -------------------------                                           *
+*                                                                      *
+*  Covered Software is provided under this License on an "as is"       *
+*  basis, without warranty of any kind, either expressed, implied, or  *
+*  statutory, including, without limitation, warranties that the       *
+*  Covered Software is free of defects, merchantable, fit for a        *
+*  particular purpose or non-infringing. The entire risk as to the     *
+*  quality and performance of the Covered Software is with You.        *
+*  Should any Covered Software prove defective in any respect, You     *
+*  (not any Contributor) assume the cost of any necessary servicing,   *
+*  repair, or correction. This disclaimer of warranty constitutes an   *
+*  essential part of this License. No use of any Covered Software is   *
+*  authorized under this License except under this disclaimer.         *
+*                                                                      *
+************************************************************************
+
+************************************************************************
+*                                                                      *
+*  7. Limitation of Liability                                          *
+*  --------------------------                                          *
+*                                                                      *
+*  Under no circumstances and under no legal theory, whether tort      *
+*  (including negligence), contract, or otherwise, shall any           *
+*  Contributor, or anyone who distributes Covered Software as          *
+*  permitted above, be liable to You for any direct, indirect,         *
+*  special, incidental, or consequential damages of any character      *
+*  including, without limitation, damages for lost profits, loss of    *
+*  goodwill, work stoppage, computer failure or malfunction, or any    *
+*  and all other commercial damages or losses, even if such party      *
+*  shall have been informed of the possibility of such damages. This   *
+*  limitation of liability shall not apply to liability for death or   *
+*  personal injury resulting from such party's negligence to the       *
+*  extent applicable law prohibits such limitation. Some               *
+*  jurisdictions do not allow the exclusion or limitation of           *
+*  incidental or consequential damages, so this exclusion and          *
+*  limitation may not apply to You.                                    *
+*                                                                      *
+************************************************************************
+
+8. Litigation
+-------------
+
+Any litigation relating to this License may be brought only in the
+courts of a jurisdiction where the defendant maintains its principal
+place of business and such litigation shall be governed by laws of that
+jurisdiction, without reference to its conflict-of-law provisions.
+Nothing in this Section shall prevent a party's ability to bring
+cross-claims or counter-claims.
+
+9. Miscellaneous
+----------------
+
+This License represents the complete agreement concerning the subject
+matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent
+necessary to make it enforceable. Any law or regulation which provides
+that the language of a contract shall be construed against the drafter
+shall not be used to construe this License against a Contributor.
+
+10. Versions of the License
+---------------------------
+
+10.1. New Versions
+
+Mozilla Foundation is the license steward. Except as provided in Section
+10.3, no one other than the license steward has the right to modify or
+publish new versions of this License. Each version will be given a
+distinguishing version number.
+
+10.2. Effect of New Versions
+
+You may distribute the Covered Software under the terms of the version
+of the License under which You originally received the Covered Software,
+or under the terms of any subsequent version published by the license
+steward.
+
+10.3. Modified Versions
+
+If you create software not governed by this License, and you want to
+create a new license for such software, you may create and use a
+modified version of this License if you rename the license and remove
+any references to the name of the license steward (except to note that
+such modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary
+Licenses
+
+If You choose to distribute Source Code Form that is Incompatible With
+Secondary Licenses under the terms of this version of the License, the
+notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+-------------------------------------------
+
+  This Source Code Form is subject to the terms of the Mozilla Public
+  License, v. 2.0. If a copy of the MPL was not distributed with this
+  file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular
+file, then You may include the notice in a location (such as a LICENSE
+file in a relevant directory) where a recipient would be likely to look
+for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - "Incompatible With Secondary Licenses" Notice
+---------------------------------------------------------
+
+  This Source Code Form is "Incompatible With Secondary Licenses", as
+  defined by the Mozilla Public License, v. 2.0.
diff --git a/Makefile.am b/Makefile.am
new file mode 100755
index 0000000..c523f31
--- /dev/null
+++ b/Makefile.am
@@ -0,0 +1,40 @@
+ACLOCAL_AMFLAGS = -I m4
+
+if DM
+D = src
+else
+D =
+endif
+
+if WEB
+W = www
+else
+W =
+endif
+
+SUBDIRS = conf tools $(W) $(D)
+
+MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
+
+hh_web_conf_dir    = hedgehog/conf
+hh_web_source_dir  = hedgehog/www
+hh_data_dir        = hedgehog/data
+hh_plotcache_dir   = hedgehog/www/plotcache
+hh_rssac_dir       = hedgehog/www/rssac
+hh_conf_dir        = hedgehog
+hh_conf_file       = hedgehog.yaml
+
+# Used to substitute into the hedgehog.yaml conf file and the hedgehog.conf file
+subst_HH_DIRS = sed -e "s|@HH_WEB_CONF@|@datadir@/$(hh_web_conf_dir)|" \
+					-e "s|@HH_WEB_SOURCE@|@datadir@/$(hh_web_source_dir)|" \
+					-e "s|@HH_DATA_DIR@|@localstatedir@/$(hh_data_dir)|" \
+					-e "s|@HH_PLOTCACHE@|@localstatedir@/$(hh_plotcache_dir)|" \
+					-e "s|@HH_CONF@|@sysconfdir@/$(hh_conf_dir)|" \
+					-e "s|@HH_RSSAC@|@localstatedir@/$(hh_rssac_dir)|"
+
+#Used to substitute into startupR.in and hedgeghog_conf_read.sh
+subst_HH_CONF_FILE = sed -e "s|@HH_CONF_FILE@|@sysconfdir@/$(hh_conf_dir)/$(hh_conf_file)|" 
+
+export subst_HH_DIRS
+export subst_HH_CONF_FILE
+export hh_web_conf_dir
diff --git a/README b/README
new file mode 100755
index 0000000..3b51d81
--- /dev/null
+++ b/README
@@ -0,0 +1,55 @@
+Hedgehog
+
+DESCRIPTION
+-----------
+*Please note that Hedgehog is no longer actively developed.* 
+The DNS-STATS project is now focussing on
+a newer data collection (dns-stats-compactor) and presentation solution - see the
+http://www.dns-stats.org website.
+
+Hedgehog is a visualisation tool for DNS statistics that consumes data
+acquired with the DSC collector.
+
+Hedgehog was initially developed for ICANN by Sinodun IT and is now released
+open source. This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0. If a copy of the MPL was not distributed with this
+file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+
+REGISTRATION
+-------------
+If you use Hedgehog, please consider subscribing to the Users Mailing list:
+https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
+and providing feedback. It helps us to know how Hedgehog is being used!
+
+OVERVIEW
+--------
+For general information and the User guide see the Hedgehog website:
+https://github.com/dns-stats/hedgehog/wiki
+
+USER GUIDE
+----------
+Read the User_Guide.pdf in the docs directory for the lastest version or look
+online: https://github.com/dns-stats/hedgehog/wiki/Documentation
+
+PACKAGES
+---------
+The latest Hedgehog packages are available from ppa:dns-stats/presenter
+
+SOURCE INSTALLATION
+------------
+This is no longer officially supported.
+
+VAGRANT INSTALLATION
+------------
+You can load Hedgehog locally using a virtual machine and Vagrant 
+(https://www.vagrantup.com/downloads) â Requires also VirtualBox.
+Execute `vagrant up` to build the vm and run provisionning scripts 
+(dependencies installation, compilation, configuration).
+Once completed, you can access to the server using this url: 
+`http://localhost:8080/hedgehog`
+
+SUPPORT
+-------
+- Mailing list: https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
+- Issue tracker: https://github.com/dns-stats/hedgehog/issues
diff --git a/Vagrantfile b/Vagrantfile
new file mode 100644
index 0000000..ca4d76d
--- /dev/null
+++ b/Vagrantfile
@@ -0,0 +1,22 @@
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+  config.vm.box = "ubuntu/trusty64"
+  config.vm.box_check_update = false
+
+  config.vm.network :forwarded_port, guest: 80, host: 8080, host_ip: "127.0.0.1"
+  config.vm.network :private_network, ip: '192.168.50.50'
+
+  config.vm.provider "virtualbox" do |v|
+    v.memory = 2048
+    v.cpus = 2
+  end
+
+  config.vm.synced_folder '.', '/vagrant'
+
+  config.vm.provision "shell", path: "provision/dependencies.sh"
+  config.vm.provision "shell", path: "provision/compile.sh"
+  config.vm.provision "shell", path: "provision/launch.sh"
+
+end
diff --git a/autogen.sh b/autogen.sh
new file mode 100755
index 0000000..5515a26
--- /dev/null
+++ b/autogen.sh
@@ -0,0 +1,7 @@
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+
+autoreconf --install --force
diff --git a/conf/Makefile.am b/conf/Makefile.am
new file mode 100755
index 0000000..0da3f7f
--- /dev/null
+++ b/conf/Makefile.am
@@ -0,0 +1,27 @@
+CLEANFILES = $(localstate_DATA) $(noinst_DATA)
+
+sysconfdir = @sysconfdir@/hedgehog
+
+nobase_sysconf_DATA = $(srcdir)/tlds.d/cctlds.csv \
+                      $(srcdir)/tlds.d/idn_cctlds.csv \
+                      $(srcdir)/tlds.d/legacy_gtlds.csv \
+                      $(srcdir)/tlds.d/new_gtlds-20131023-20150611.csv \
+                      $(srcdir)/tlds.d/new_gtlds-20150614-20151223.csv
+
+noinst_DATA = hedgehog.yaml
+
+hedgehog.yaml: $(srcdir)/hedgehog.yaml.in Makefile
+	$(subst_HH_DIRS) < $(srcdir)/hedgehog.yaml.in > hedgehog.yaml
+
+EXTRA_DIST = (srcdir)/hedgehog.yaml.in
+
+install-data-hook:
+	test -d ${DESTDIR}${sysconfdir} || mkdir -p ${DESTDIR}${sysconfdir}
+	test -f ${DESTDIR}${sysconfdir}/hedgehog.yaml || \
+		${INSTALL_DATA} -m 644 hedgehog.yaml ${DESTDIR}${sysconfdir}
+	${INSTALL_DATA} -m 644 hedgehog.yaml ${DESTDIR}${sysconfdir}/hedgehog.yaml.sample
+	test -f ${DESTDIR}${sysconfdir}/nodes.csv || \
+		${INSTALL_DATA} -m 644 $(srcdir)/nodes.csv ${DESTDIR}${sysconfdir}
+	test -f ${DESTDIR}${sysconfdir}/service_addr.csv || \
+		${INSTALL_DATA} -m 644 $(srcdir)/service_addr.csv ${DESTDIR}${sysconfdir}
+	${INSTALL_DATA} -m 644 $(srcdir)/nodes.csv ${DESTDIR}${sysconfdir}/nodes.csv.sample
\ No newline at end of file
diff --git a/conf/hedgehog.yaml.in b/conf/hedgehog.yaml.in
new file mode 100755
index 0000000..e202ba4
--- /dev/null
+++ b/conf/hedgehog.yaml.in
@@ -0,0 +1,33 @@
+## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+##
+## This Source Code Form is subject to the terms of the Mozilla Public
+## License, v. 2.0. If a copy of the MPL was not distributed with this
+## file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+## Developed by Sinodun IT (www.sinodun.com)
+
+## YAML config for Hedgehog
+--- 
+# Database parameters to be specified by the user - these must not be defaulted
+database:
+  host         : /var/run/postgresql # specify a host for the postgresql DB. If this begins with a slash,
+                                     # it specifies the directory in which the socket file is stored.
+  port         : 5432      # specify port for the postgresql DB.
+  name         : hedgehog  # specify dbname for the postgresql DB.
+  owner        : hedgehog  # specify a user to own the postgresql DB.
+                           # [Required for Data Manager component]
+  owner_pass   :           # specify a password for the owner user if needed.
+  read_user    : www-data  # specify a read user for the postgresql DB.
+                           # [Required for Web front-end component]
+  read_pass    :           # specify a password for the read user if needed.
+    
+# Directory structure
+# !! This information is currently auto-populated at compile time and it is _not_ recommended that users alter this !!
+directories:
+  conf         : @HH_CONF@          # directory of hedgehog configuration files
+  data         : @HH_DATA_DIR@      # directory holding the XML for all the servers
+  plotcache    : @HH_PLOTCACHE@     # location of the plotcache used by the web front end to hold image files
+  rssac        : @HH_RSSAC@         # location of the rssac directory where the daily reports are stored
+  web_conf     : @HH_WEB_CONF@      # configuration files for web front end
+  web_source   : @HH_WEB_SOURCE@    # source files for web front end
+...
diff --git a/conf/nodes.csv b/conf/nodes.csv
new file mode 100644
index 0000000..fef17c7
--- /dev/null
+++ b/conf/nodes.csv
@@ -0,0 +1,54 @@
+######################### NODES CONFIGURATION FILE ###################
+# Format is:     server-name, node-name, [node-group],[node-country],
+#                [node-city],[node-instance],[node-ip]
+# Server name, node name, group and instance fields can contain alphanumeric
+# characters, full stops (.) and hyphens (-) only.
+# Country and city fields can contain alphanumeric characters, spaces and
+# hyphens (-) only.
+# Nodes may be grouped and plotted by Country,City or Instance in 2.1.
+# '#' is the comment character
+# 
+# server-name   - This must be unique and it must match the name used in the
+#                 data directory path for uploaded files.
+# node-name     - This must be unique for a given server and it must 
+#                 match the name used in the data directory path for 
+#                 uploaded files.
+# node-group    - This is used in the node grouping tabs in the Hedgehog
+#                 front end. It should be unique for a given server. 
+#                 It is optional and if not
+#                 specified the node is added to a group called 'Other'.
+# node-country  - This is the country that the node is located in
+#                 It is optional and if not specified 'No Country' is used.
+# node-city     - This is the city that the node is located in
+#                 It is optional and if not specified 'No City' is used.
+#                 This is used to plot node locations in GEO plots
+# node-instance - This is used to collect nodes into subgroups called 'Instances' 
+#                 (e.g. nodes within the same city) for convenient selection
+#                 and plotting. It should be unique for a given group. It is 
+#                 optional and if not specified 'No Instance' is used.
+# node-ip       - The management IP address of the node (IPv4 or IPv6)
+#                 It is optional and if not specified is left blank. This is 
+#                 required if RSSAC load-time/zone-size data is to be collected.
+# rename-or-remove - This field is used to either rename or remove a node. It 
+#                    is optional. It is only honored if the -r flag is set and
+#                    must be empty if the -r flag is not used. 
+#                    If this field contains the name of an existing node and the
+#                    node-name contains a different name then the node is
+#                    renamed. 
+#                    If this field contains "remove" then the node and all 
+#                    associated data is removed from the database. Data on disk
+#                    is not touched.
+#
+# Example input: (uncomment the following)
+#
+#Server-A,Node-1,Region-1,Country-1,City-1,Instance-1,10.0.0.2,Node-a
+#Server-A,Node-2,Region-1,Country-1,City-2,Instance-2,
+#Server-A,Node-3,Region-1,Country-2,City-3,Instance-3,
+#Server-A,Node-4,Region-1,Country-2,City-3,Instance-3,
+#Server-A,Node-5,Region-2,Country-3,City-4,Instance-4,
+#Server-A,Node-6,Region-2,Country-3,City-4,Instance-4,, remove
+#Server-A,Node-7,Region-3,Country-3,City-4,Instance-5,10.0.0.3
+#Server-B,Node-1,Region-1,Country-4,,,
+#Server-B,Node-2,Region-2,,,Instance-6,
+#Server-C,Node-5
+######################################################################
diff --git a/conf/service_addr.csv b/conf/service_addr.csv
new file mode 100644
index 0000000..b623b74
--- /dev/null
+++ b/conf/service_addr.csv
@@ -0,0 +1,6 @@
+######################### SERVICE ADDRESS CONFIGURATION FILE ###################
+# Format is:     server-name, address (IPv4 or IPv6)
+#
+# L-Root,2001:500:9f::42
+# L-Root,199.7.83.42
+# L-Root,2001:500:3::42
diff --git a/conf/tlds.d/cctlds.csv b/conf/tlds.d/cctlds.csv
new file mode 100644
index 0000000..5e85bc0
--- /dev/null
+++ b/conf/tlds.d/cctlds.csv
@@ -0,0 +1,250 @@
+# Format is: alabel, type
+
+ac, ccTLD
+ad, ccTLD
+ae, ccTLD
+af, ccTLD
+ag, ccTLD
+ai, ccTLD
+al, ccTLD
+am, ccTLD
+an, ccTLD
+ao, ccTLD
+aq, ccTLD
+ar, ccTLD
+as, ccTLD
+at, ccTLD
+au, ccTLD
+aw, ccTLD
+ax, ccTLD
+az, ccTLD
+ba, ccTLD
+bb, ccTLD
+bd, ccTLD
+be, ccTLD
+bf, ccTLD
+bg, ccTLD
+bh, ccTLD
+bi, ccTLD
+bj, ccTLD
+bm, ccTLD
+bn, ccTLD
+bo, ccTLD
+br, ccTLD
+bs, ccTLD
+bt, ccTLD
+bv, ccTLD
+bw, ccTLD
+by, ccTLD
+bz, ccTLD
+ca, ccTLD
+cc, ccTLD
+cd, ccTLD
+cf, ccTLD
+cg, ccTLD
+ch, ccTLD
+ci, ccTLD
+ck, ccTLD
+cl, ccTLD
+cm, ccTLD
+cn, ccTLD
+co, ccTLD
+cr, ccTLD
+cu, ccTLD
+cv, ccTLD
+cw, ccTLD
+cx, ccTLD
+cy, ccTLD
+cz, ccTLD
+de, ccTLD
+dj, ccTLD
+dk, ccTLD
+dm, ccTLD
+do, ccTLD
+dz, ccTLD
+ec, ccTLD
+ee, ccTLD
+eg, ccTLD
+er, ccTLD
+es, ccTLD
+et, ccTLD
+eu, ccTLD
+fi, ccTLD
+fj, ccTLD
+fk, ccTLD
+fm, ccTLD
+fo, ccTLD
+fr, ccTLD
+ga, ccTLD
+gb, ccTLD
+gd, ccTLD
+ge, ccTLD
+gf, ccTLD
+gg, ccTLD
+gh, ccTLD
+gi, ccTLD
+gl, ccTLD
+gm, ccTLD
+gn, ccTLD
+gp, ccTLD
+gq, ccTLD
+gr, ccTLD
+gs, ccTLD
+gt, ccTLD
+gu, ccTLD
+gw, ccTLD
+gy, ccTLD
+hk, ccTLD
+hm, ccTLD
+hn, ccTLD
+hr, ccTLD
+ht, ccTLD
+hu, ccTLD
+id, ccTLD
+ie, ccTLD
+il, ccTLD
+im, ccTLD
+in, ccTLD
+io, ccTLD
+iq, ccTLD
+ir, ccTLD
+is, ccTLD
+it, ccTLD
+je, ccTLD
+jm, ccTLD
+jo, ccTLD
+jp, ccTLD
+ke, ccTLD
+kg, ccTLD
+kh, ccTLD
+ki, ccTLD
+km, ccTLD
+kn, ccTLD
+kp, ccTLD
+kr, ccTLD
+kw, ccTLD
+ky, ccTLD
+kz, ccTLD
+la, ccTLD
+lb, ccTLD
+lc, ccTLD
+li, ccTLD
+lk, ccTLD
+lr, ccTLD
+ls, ccTLD
+lt, ccTLD
+lu, ccTLD
+lv, ccTLD
+ly, ccTLD
+ma, ccTLD
+mc, ccTLD
+md, ccTLD
+me, ccTLD
+mg, ccTLD
+mh, ccTLD
+mk, ccTLD
+ml, ccTLD
+mm, ccTLD
+mn, ccTLD
+mo, ccTLD
+mp, ccTLD
+mq, ccTLD
+mr, ccTLD
+ms, ccTLD
+mt, ccTLD
+mu, ccTLD
+mv, ccTLD
+mw, ccTLD
+mx, ccTLD
+my, ccTLD
+mz, ccTLD
+na, ccTLD
+nc, ccTLD
+ne, ccTLD
+nf, ccTLD
+ng, ccTLD
+ni, ccTLD
+nl, ccTLD
+no, ccTLD
+np, ccTLD
+nr, ccTLD
+nu, ccTLD
+nz, ccTLD
+om, ccTLD
+pa, ccTLD
+pe, ccTLD
+pf, ccTLD
+pg, ccTLD
+ph, ccTLD
+pk, ccTLD
+pl, ccTLD
+pm, ccTLD
+pn, ccTLD
+pr, ccTLD
+ps, ccTLD
+pt, ccTLD
+pw, ccTLD
+py, ccTLD
+qa, ccTLD
+re, ccTLD
+ro, ccTLD
+rs, ccTLD
+ru, ccTLD
+rw, ccTLD
+sa, ccTLD
+sb, ccTLD
+sc, ccTLD
+sd, ccTLD
+se, ccTLD
+sg, ccTLD
+sh, ccTLD
+si, ccTLD
+sj, ccTLD
+sk, ccTLD
+sl, ccTLD
+sm, ccTLD
+sn, ccTLD
+so, ccTLD
+sr, ccTLD
+st, ccTLD
+su, ccTLD
+sv, ccTLD
+sx, ccTLD
+sy, ccTLD
+sz, ccTLD
+tc, ccTLD
+td, ccTLD
+tf, ccTLD
+tg, ccTLD
+th, ccTLD
+tj, ccTLD
+tk, ccTLD
+tl, ccTLD
+tm, ccTLD
+tn, ccTLD
+to, ccTLD
+tr, ccTLD
+tt, ccTLD
+tv, ccTLD
+tw, ccTLD
+tz, ccTLD
+ua, ccTLD
+ug, ccTLD
+uk, ccTLD
+us, ccTLD
+uy, ccTLD
+uz, ccTLD
+va, ccTLD
+vc, ccTLD
+ve, ccTLD
+vg, ccTLD
+vi, ccTLD
+vn, ccTLD
+vu, ccTLD
+wf, ccTLD
+ws, ccTLD
+ye, ccTLD
+yt, ccTLD
+za, ccTLD
+zm, ccTLD
+zw, ccTLD
\ No newline at end of file
diff --git a/conf/tlds.d/idn_cctlds.csv b/conf/tlds.d/idn_cctlds.csv
new file mode 100644
index 0000000..f0497f8
--- /dev/null
+++ b/conf/tlds.d/idn_cctlds.csv
@@ -0,0 +1,41 @@
+# Format is: alabel, type
+
+xn--3e0b707e, ccTLD
+xn--45brj9c, ccTLD
+xn--80ao21a, ccTLD
+xn--90a3ac, ccTLD
+xn--90ais, ccTLD
+xn--clchc0ea0b2g2a9gcd, ccTLD
+xn--d1alf, ccTLD
+xn--fiqs8s, ccTLD
+xn--fiqz9s, ccTLD
+xn--fpcrj9c3d, ccTLD
+xn--fzc2c9e2c, ccTLD
+xn--gecrj9c, ccTLD
+xn--h2brj9c, ccTLD
+xn--j1amh, ccTLD
+xn--j6w193g, ccTLD
+xn--kprw13d, ccTLD
+xn--kpry57d, ccTLD
+xn--l1acc, ccTLD
+xn--lgbbat1ad8j, ccTLD
+xn--mgb9awbf, ccTLD
+xn--mgba3a4f16a, ccTLD
+xn--mgbaam7a8h, ccTLD
+xn--mgbayh7gpa, ccTLD
+xn--mgbbh1a71e, ccTLD
+xn--mgbc0a9azcg, ccTLD
+xn--mgberp4a5d4ar, ccTLD
+xn--mgbx4cd0ab, ccTLD
+xn--node, ccTLD
+xn--o3cw4h, ccTLD
+xn--ogbpf8fl, ccTLD
+xn--p1ai, ccTLD
+xn--pgbs0dh, ccTLD
+xn--s9brj9c, ccTLD
+xn--wgbh1c, ccTLD
+xn--wgbl6a, ccTLD
+xn--xkc2al3hye2a, ccTLD
+xn--xkc2dl3a5ee0h, ccTLD
+xn--yfro4i67o, ccTLD
+xn--ygbi2ammx, ccTLD
\ No newline at end of file
diff --git a/conf/tlds.d/legacy_gtlds.csv b/conf/tlds.d/legacy_gtlds.csv
new file mode 100644
index 0000000..8e8b2bd
--- /dev/null
+++ b/conf/tlds.d/legacy_gtlds.csv
@@ -0,0 +1,26 @@
+# Format is: alabel, type
+
+aero, Legacy-gTLD
+arpa, Legacy-gTLD
+asia, Legacy-gTLD
+biz, Legacy-gTLD
+cat, Legacy-gTLD
+com, Legacy-gTLD
+coop, Legacy-gTLD
+edu, Legacy-gTLD
+gov, Legacy-gTLD
+info, Legacy-gTLD
+int, Legacy-gTLD
+jobs, Legacy-gTLD
+mil, Legacy-gTLD
+mobi, Legacy-gTLD
+museum, Legacy-gTLD
+name, Legacy-gTLD
+net, Legacy-gTLD
+org, Legacy-gTLD
+post, Legacy-gTLD
+pro, Legacy-gTLD
+sncf, Legacy-gTLD
+tel, Legacy-gTLD
+travel, Legacy-gTLD
+xxx, Legacy-gTLD
\ No newline at end of file
diff --git a/conf/tlds.d/new_gtlds-20131023-20150611.csv b/conf/tlds.d/new_gtlds-20131023-20150611.csv
new file mode 100644
index 0000000..fdd04fa
--- /dev/null
+++ b/conf/tlds.d/new_gtlds-20131023-20150611.csv
@@ -0,0 +1,656 @@
+# Format is: alabel, type
+
+LASALLE, New-gTLD
+JLC, New-gTLD
+MICROSOFT, New-gTLD
+HOTMAIL, New-gTLD
+WINDOWS, New-gTLD
+BING, New-gTLD
+AZURE, New-gTLD
+MONTBLANC, New-gTLD
+HOMEDEPOT, New-gTLD
+XBOX, New-gTLD
+BIBLE, New-gTLD
+SKI, New-gTLD
+xn--imr513n, New-gTLD
+WALTER, New-gTLD
+SANDVIK, New-gTLD
+SANDVIKCOROMANT, New-gTLD
+BBVA, New-gTLD
+MBA, New-gTLD
+JLL, New-gTLD
+FYI, New-gTLD
+THD, New-gTLD
+MEN, New-gTLD
+LUPIN, New-gTLD
+CORSICA, New-gTLD
+CISCO, New-gTLD
+EARTH, New-gTLD
+SOCCER, New-gTLD
+COUPONS, New-gTLD
+ICBC, New-gTLD
+BROTHER, New-gTLD
+ACCENTURE, New-gTLD
+PHILIPS, New-gTLD
+xn--fjq720a, New-gTLD
+NEC, New-gTLD
+TAXI, New-gTLD
+xn--estv75g, New-gTLD
+HOCKEY, New-gTLD
+RUN, New-gTLD
+xn--y9a3aq, New-gTLD
+xn--mgbpl2fh, New-gTLD
+THEATER, New-gTLD
+AIG, New-gTLD
+ICU, New-gTLD
+LOL, New-gTLD
+LIAISON, New-gTLD
+CFA, New-gTLD
+NADEX, New-gTLD
+CARS, New-gTLD
+AUTO, New-gTLD
+TORAY, New-gTLD
+HITACHI, New-gTLD
+BRIDGESTONE, New-gTLD
+SENER, New-gTLD
+RENT, New-gTLD
+HONDA, New-gTLD
+SWISS, New-gTLD
+DOG, New-gTLD
+BROKER, New-gTLD
+ABB, New-gTLD
+SEX, New-gTLD
+SEAT, New-gTLD
+WEIR, New-gTLD
+JEWELRY, New-gTLD
+XEROX, New-gTLD
+SHOW, New-gTLD
+TEAM, New-gTLD
+SONY, New-gTLD
+xn--kcrx77d1x4a, New-gTLD
+EXPRESS, New-gTLD
+BAUHAUS, New-gTLD
+CAFE, New-gTLD
+CYOU, New-gTLD
+RACING, New-gTLD
+xn--nyqy26a, New-gTLD
+SCHOLARSHIPS, New-gTLD
+LOVE, New-gTLD
+xn--30rr7y, New-gTLD
+MMA, New-gTLD
+AFL, New-gTLD
+REDSTONE, New-gTLD
+xn--9et52u, New-gTLD
+BOND, New-gTLD
+SAP, New-gTLD
+KOMATSU, New-gTLD
+DOHA, New-gTLD
+MTN, New-gTLD
+REVIEW, New-gTLD
+WIN, New-gTLD
+PANERAI, New-gTLD
+DOWNLOAD, New-gTLD
+MOVIE, New-gTLD
+FAITH, New-gTLD
+TICKETS, New-gTLD
+ACCOUNTANT, New-gTLD
+LOAN, New-gTLD
+DATE, New-gTLD
+TOURS, New-gTLD
+ADS, New-gTLD
+PLUS, New-gTLD
+FILM, New-gTLD
+GOLD, New-gTLD
+GOLF, New-gTLD
+GUGE, New-gTLD
+BBC, New-gTLD
+TECH, New-gTLD
+NEWS, New-gTLD
+xn--vuq861b, New-gTLD
+PAGE, New-gTLD
+PIAGET, New-gTLD
+SITE, New-gTLD
+FAN, New-gTLD
+ONLINE, New-gTLD
+CFD, New-gTLD
+TRADING, New-gTLD
+SPREADBETTING, New-gTLD
+ERNI, New-gTLD
+MARKETS, New-gTLD
+FOREX, New-gTLD
+CHLOE, New-gTLD
+XIN, New-gTLD
+PICTET, New-gTLD
+ABBOTT, New-gTLD
+NISSAN, New-gTLD
+INFINITI, New-gTLD
+DATSUN, New-gTLD
+MTPC, New-gTLD
+ORACLE, New-gTLD
+GOO, New-gTLD
+LECLERC, New-gTLD
+JAVA, New-gTLD
+EPSON, New-gTLD
+MAIF, New-gTLD
+xn--mxtq1m, New-gTLD
+SUCKS, New-gTLD
+STUDY, New-gTLD
+BOATS, New-gTLD
+COURSES, New-gTLD
+CASINO, New-gTLD
+FANS, New-gTLD
+FOOTBALL, New-gTLD
+GOLDPOINT, New-gTLD
+SCHOOL, New-gTLD
+YODOBASHI, New-gTLD
+CBN, New-gTLD
+GDN, New-gTLD
+APARTMENTS, New-gTLD
+SAXO, New-gTLD
+NICO, New-gTLD
+TENNIS, New-gTLD
+BINGO, New-gTLD
+CANON, New-gTLD
+STYLE, New-gTLD
+TOSHIBA, New-gTLD
+CHAT, New-gTLD
+NTT, New-gTLD
+KYOTO, New-gTLD
+DCLK, New-gTLD
+xn--b4w605ferd, New-gTLD
+IFM, New-gTLD
+GOOG, New-gTLD
+TEMASEK, New-gTLD
+HANGOUT, New-gTLD
+BARCLAYS, New-gTLD
+DESIGN, New-gTLD
+DABUR, New-gTLD
+HERMES, New-gTLD
+BARCLAYCARD, New-gTLD
+JCB, New-gTLD
+ONE, New-gTLD
+LOTTE, New-gTLD
+MARRIOTT, New-gTLD
+FIT, New-gTLD
+LAT, New-gTLD
+KDDI, New-gTLD
+BANK, New-gTLD
+SHRIRAM, New-gTLD
+ZUERICH, New-gTLD
+FLOWERS, New-gTLD
+GGEE, New-gTLD
+SALE, New-gTLD
+AMSTERDAM, New-gTLD
+VIDEO, New-gTLD
+DOCS, New-gTLD
+TIRES, New-gTLD
+DEV, New-gTLD
+SCHWARZ, New-gTLD
+IWC, New-gTLD
+OSAKA, New-gTLD
+SEW, New-gTLD
+GARDEN, New-gTLD
+LIDL, New-gTLD
+DOOSAN, New-gTLD
+SKY, New-gTLD
+CARTIER, New-gTLD
+SAMSUNG, New-gTLD
+xn--czrs0t, New-gTLD
+ADULT, New-gTLD
+PORN, New-gTLD
+TRUST, New-gTLD
+EUROVISION, New-gTLD
+FASHION, New-gTLD
+LATROBE, New-gTLD
+IRISH, New-gTLD
+xn--hxt814e, New-gTLD
+AQUARELLE, New-gTLD
+MEMORIAL, New-gTLD
+LEGAL, New-gTLD
+MONEY, New-gTLD
+COACH, New-gTLD
+EVERBANK, New-gTLD
+xn--flw351e, New-gTLD
+MADRID, New-gTLD
+FIRMDALE, New-gTLD
+xn--qcka1pmc, New-gTLD
+MORMON, New-gTLD
+LDS, New-gTLD
+PARTY, New-gTLD
+xn--45q11c, New-gTLD
+CRICKET, New-gTLD
+SCIENCE, New-gTLD
+REIT, New-gTLD
+ANDROID, New-gTLD
+SYDNEY, New-gTLD
+BLOOMBERG, New-gTLD
+ENERGY, New-gTLD
+DELIVERY, New-gTLD
+TAIPEI, New-gTLD
+EMERCK, New-gTLD
+FLSMIDTH, New-gTLD
+BAND, New-gTLD
+YOGA, New-gTLD
+CRS, New-gTLD
+ABOGADO, New-gTLD
+RIP, New-gTLD
+WEDDING, New-gTLD
+POKER, New-gTLD
+ALSACE, New-gTLD
+ALLFINANZ, New-gTLD
+IBM, New-gTLD
+FORSALE, New-gTLD
+xn--vermgensberatung-pwb, New-gTLD
+xn--vermgensberater-ctb, New-gTLD
+xn--p1acf, New-gTLD
+TUI, New-gTLD
+DVAG, New-gTLD
+POHL, New-gTLD
+WORK, New-gTLD
+CASA, New-gTLD
+BUDAPEST, New-gTLD
+WORLD, New-gTLD
+GOOGLE, New-gTLD
+FLY, New-gTLD
+NEXUS, New-gTLD
+CHANNEL, New-gTLD
+PROF, New-gTLD
+GLE, New-gTLD
+ZIP, New-gTLD
+CAL, New-gTLD
+CHROME, New-gTLD
+WME, New-gTLD
+PHARMACY, New-gTLD
+GMX, New-gTLD
+BOO, New-gTLD
+DAD, New-gTLD
+DAY, New-gTLD
+FRL, New-gTLD
+ING, New-gTLD
+NEW, New-gTLD
+MOV, New-gTLD
+RSVP, New-gTLD
+EAT, New-gTLD
+MEME, New-gTLD
+YOUTUBE, New-gTLD
+HERE, New-gTLD
+PROD, New-gTLD
+ESQ, New-gTLD
+IMMO, New-gTLD
+PIZZA, New-gTLD
+GMAIL, New-gTLD
+GBIZ, New-gTLD
+OTSUKA, New-gTLD
+BUSINESS, New-gTLD
+xn--vhquv, New-gTLD
+NETWORK, New-gTLD
+CLICK, New-gTLD
+DIET, New-gTLD
+HOW, New-gTLD
+OOO, New-gTLD
+UOL, New-gTLD
+HELP, New-gTLD
+HOSTING, New-gTLD
+PROPERTY, New-gTLD
+LTDA, New-gTLD
+CERN, New-gTLD
+CARAVAN, New-gTLD
+SCA, New-gTLD
+xn--xhq521b, New-gTLD
+xn--1qqw23a, New-gTLD
+BNPPARIBAS, New-gTLD
+RESTAURANT, New-gTLD
+CYMRU, New-gTLD
+SARL, New-gTLD
+GIFTS, New-gTLD
+WALES, New-gTLD
+TATAR, New-gTLD
+TOP, New-gTLD
+HEALTHCARE, New-gTLD
+REALTOR, New-gTLD
+WILLIAMHILL, New-gTLD
+ONG, New-gTLD
+PRAXI, New-gTLD
+YANDEX, New-gTLD
+AUCTION, New-gTLD
+WHOSWHO, New-gTLD
+KRD, New-gTLD
+NGO, New-gTLD
+NRA, New-gTLD
+SPIEGEL, New-gTLD
+LACAIXA, New-gTLD
+LGBT, New-gTLD
+GENT, New-gTLD
+NRW, New-gTLD
+SCB, New-gTLD
+DEALS, New-gTLD
+MELBOURNE, New-gTLD
+CITY, New-gTLD
+SCHMIDT, New-gTLD
+CUISINELLA, New-gTLD
+CANCERRESEARCH, New-gTLD
+DIRECT, New-gTLD
+PLACE, New-gTLD
+SUZUKI, New-gTLD
+ACTIVE, New-gTLD
+MINI, New-gTLD
+BMW, New-gTLD
+OVH, New-gTLD
+CAPETOWN, New-gTLD
+PHYSIO, New-gTLD
+LOTTO, New-gTLD
+DURBAN, New-gTLD
+GREEN, New-gTLD
+JOBURG, New-gTLD
+VLAANDEREN, New-gTLD
+BRUSSELS, New-gTLD
+SURF, New-gTLD
+xn--kput3i, New-gTLD
+BZH, New-gTLD
+SCOT, New-gTLD
+ORGANIC, New-gTLD
+GLOBAL, New-gTLD
+ENGINEER, New-gTLD
+NHK, New-gTLD
+HAMBURG, New-gTLD
+REHAB, New-gTLD
+TIROL, New-gTLD
+REPUBLICAN, New-gTLD
+GIVES, New-gTLD
+NAVY, New-gTLD
+ARMY, New-gTLD
+BIO, New-gTLD
+LAWYER, New-gTLD
+VET, New-gTLD
+MORTGAGE, New-gTLD
+SOFTWARE, New-gTLD
+HOST, New-gTLD
+MARKET, New-gTLD
+DENTIST, New-gTLD
+HIV, New-gTLD
+ATTORNEY, New-gTLD
+PRESS, New-gTLD
+DEGREE, New-gTLD
+SPACE, New-gTLD
+WEBSITE, New-gTLD
+xn--4gbrim, New-gTLD
+YACHTS, New-gTLD
+RIO, New-gTLD
+HOMES, New-gTLD
+AUTOS, New-gTLD
+VERSICHERUNG, New-gTLD
+REISE, New-gTLD
+MOTORCYCLES, New-gTLD
+xn--czr694b, New-gTLD
+GUIDE, New-gTLD
+AUDIO, New-gTLD
+HIPHOP, New-gTLD
+LOANS, New-gTLD
+CHURCH, New-gTLD
+LIFE, New-gTLD
+BEER, New-gTLD
+JUEGOS, New-gTLD
+LUXE, New-gTLD
+CREDIT, New-gTLD
+ACCOUNTANTS, New-gTLD
+DIGITAL, New-gTLD
+CLAIMS, New-gTLD
+GMO, New-gTLD
+GLOBO, New-gTLD
+BAYERN, New-gTLD
+AIRFORCE, New-gTLD
+CREDITCARD, New-gTLD
+INSURE, New-gTLD
+FINANCE, New-gTLD
+WTC, New-gTLD
+CITIC, New-gTLD
+xn--80adxhks, New-gTLD
+MOSCOW, New-gTLD
+LIMITED, New-gTLD
+FAIL, New-gTLD
+EXCHANGE, New-gTLD
+TAX, New-gTLD
+WTF, New-gTLD
+FUND, New-gTLD
+SURGERY, New-gTLD
+INVESTMENTS, New-gTLD
+FINANCIAL, New-gTLD
+GRATIS, New-gTLD
+FURNITURE, New-gTLD
+DENTAL, New-gTLD
+CARE, New-gTLD
+CASH, New-gTLD
+DISCOUNT, New-gTLD
+BLACKFRIDAY, New-gTLD
+CLINIC, New-gTLD
+FITNESS, New-gTLD
+SCHULE, New-gTLD
+FOO, New-gTLD
+SOY, New-gTLD
+PARIS, New-gTLD
+FROGANS, New-gTLD
+QUEBEC, New-gTLD
+PICTURES, New-gTLD
+UNIVERSITY, New-gTLD
+GAL, New-gTLD
+ASSOCIATES, New-gTLD
+REISEN, New-gTLD
+MEDIA, New-gTLD
+CAREER, New-gTLD
+TOWN, New-gTLD
+TOYS, New-gTLD
+LEASE, New-gTLD
+SERVICES, New-gTLD
+ENGINEERING, New-gTLD
+GRIPE, New-gTLD
+EUS, New-gTLD
+CAPITAL, New-gTLD
+DESI, New-gTLD
+FEEDBACK, New-gTLD
+COLLEGE, New-gTLD
+ROCKS, New-gTLD
+xn--ses554g, New-gTLD
+GOP, New-gTLD
+RYUKYU, New-gTLD
+YOKOHAMA, New-gTLD
+REST, New-gTLD
+SAARLAND, New-gTLD
+CONSULTING, New-gTLD
+VODKA, New-gTLD
+HAUS, New-gTLD
+COOKING, New-gTLD
+MOE, New-gTLD
+RODEO, New-gTLD
+COUNTRY, New-gTLD
+xn--czru2d, New-gTLD
+HORSE, New-gTLD
+FISHING, New-gTLD
+VEGAS, New-gTLD
+MIAMI, New-gTLD
+ARCHI, New-gTLD
+BLACK, New-gTLD
+REN, New-gTLD
+MEET, New-gTLD
+SOHU, New-gTLD
+LONDON, New-gTLD
+NYC, New-gTLD
+COLOGNE, New-gTLD
+AXA, New-gTLD
+WEBCAM, New-gTLD
+TRADE, New-gTLD
+JETZT, New-gTLD
+xn--rhqv96g, New-gTLD
+DNP, New-gTLD
+INK, New-gTLD
+xn--nqv7f, New-gTLD
+xn--i1b6b1a6a2e, New-gTLD
+xn--nqv7fs00ema, New-gTLD
+xn--c1avg, New-gTLD
+KOELN, New-gTLD
+BID, New-gTLD
+OKINAWA, New-gTLD
+VOTE, New-gTLD
+VOTO, New-gTLD
+BAR, New-gTLD
+KRED, New-gTLD
+BEST, New-gTLD
+CHRISTMAS, New-gTLD
+PUB, New-gTLD
+xn--d1acj3b, New-gTLD
+ACTOR, New-gTLD
+SUPPLIES, New-gTLD
+FISH, New-gTLD
+xn--cg4bki, New-gTLD
+VACATIONS, New-gTLD
+INDUSTRIES, New-gTLD
+SUPPLY, New-gTLD
+XYZ, New-gTLD
+WIKI, New-gTLD
+NEUSTAR, New-gTLD
+xn--mgbab2bd, New-gTLD
+MANGO, New-gTLD
+QPON, New-gTLD
+CARDS, New-gTLD
+FOUNDATION, New-gTLD
+REVIEWS, New-gTLD
+FUTBOL, New-gTLD
+VISION, New-gTLD
+CONDOS, New-gTLD
+VILLAS, New-gTLD
+PARTS, New-gTLD
+PRODUCTIONS, New-gTLD
+MAISON, New-gTLD
+xn--6frz82g, New-gTLD
+BLUE, New-gTLD
+FLIGHTS, New-gTLD
+CLEANING, New-gTLD
+REPORT, New-gTLD
+EVENTS, New-gTLD
+CRUISES, New-gTLD
+RENTALS, New-gTLD
+PARTNERS, New-gTLD
+PROPERTIES, New-gTLD
+CATERING, New-gTLD
+EXPOSED, New-gTLD
+VOTING, New-gTLD
+TOKYO, New-gTLD
+NAGOYA, New-gTLD
+DATING, New-gTLD
+COMMUNITY, New-gTLD
+BOUTIQUE, New-gTLD
+BARGAINS, New-gTLD
+TIENDA, New-gTLD
+WORKS, New-gTLD
+WED, New-gTLD
+EXPERT, New-gTLD
+WATCH, New-gTLD
+KIM, New-gTLD
+COOL, New-gTLD
+TOOLS, New-gTLD
+CLUB, New-gTLD
+BUILD, New-gTLD
+PICS, New-gTLD
+PINK, New-gTLD
+LUXURY, New-gTLD
+PHOTO, New-gTLD
+GIFT, New-gTLD
+xn--io0a7i, New-gTLD
+xn--55qx5d, New-gTLD
+MONASH, New-gTLD
+RICH, New-gTLD
+RED, New-gTLD
+SHIKSHA, New-gTLD
+xn--fiq64b, New-gTLD
+LINK, New-gTLD
+GUITARS, New-gTLD
+ZONE, New-gTLD
+CHEAP, New-gTLD
+MARKETING, New-gTLD
+DEMOCRAT, New-gTLD
+SOCIAL, New-gTLD
+AGENCY, New-gTLD
+MODA, New-gTLD
+DANCE, New-gTLD
+BERLIN, New-gTLD
+xn--3bst00m, New-gTLD
+xn--6qq986b3xl, New-gTLD
+xn--fiq228c5hs, New-gTLD
+WANG, New-gTLD
+KIWI, New-gTLD
+WIEN, New-gTLD
+EMAIL, New-gTLD
+IMMOBILIEN, New-gTLD
+xn--3ds443g, New-gTLD
+INSTITUTE, New-gTLD
+CEO, New-gTLD
+SOLAR, New-gTLD
+FARM, New-gTLD
+EDUCATION, New-gTLD
+GLASS, New-gTLD
+ONL, New-gTLD
+INTERNATIONAL, New-gTLD
+CODES, New-gTLD
+TRAINING, New-gTLD
+HOUSE, New-gTLD
+KAUFEN, New-gTLD
+NINJA, New-gTLD
+REPAIR, New-gTLD
+BUILDERS, New-gTLD
+COFFEE, New-gTLD
+FLORIST, New-gTLD
+HOLIDAY, New-gTLD
+SOLUTIONS, New-gTLD
+BUZZ, New-gTLD
+SUPPORT, New-gTLD
+RECIPES, New-gTLD
+COMPUTER, New-gTLD
+ACADEMY, New-gTLD
+CAREERS, New-gTLD
+CAB, New-gTLD
+xn--55qw42g, New-gTLD
+xn--zfr164b, New-gTLD
+SYSTEMS, New-gTLD
+DOMAINS, New-gTLD
+VIAJES, New-gTLD
+COMPANY, New-gTLD
+CAMP, New-gTLD
+LIMO, New-gTLD
+MANAGEMENT, New-gTLD
+PHOTOS, New-gTLD
+SHOES, New-gTLD
+CENTER, New-gTLD
+RUHR, New-gTLD
+MENU, New-gTLD
+UNO, New-gTLD
+xn--q9jyb4c, New-gTLD
+DIAMONDS, New-gTLD
+TIPS, New-gTLD
+PHOTOGRAPHY, New-gTLD
+DIRECTORY, New-gTLD
+ENTERPRISES, New-gTLD
+KITCHEN, New-gTLD
+TODAY, New-gTLD
+PLUMBING, New-gTLD
+GRAPHICS, New-gTLD
+CONTRACTORS, New-gTLD
+GALLERY, New-gTLD
+SEXY, New-gTLD
+CONSTRUCTION, New-gTLD
+TATTOO, New-gTLD
+TECHNOLOGY, New-gTLD
+ESTATE, New-gTLD
+LAND, New-gTLD
+BIKE, New-gTLD
+VENTURES, New-gTLD
+CAMERA, New-gTLD
+CLOTHING, New-gTLD
+LIGHTING, New-gTLD
+SINGLES, New-gTLD
+VOYAGE, New-gTLD
+GURU, New-gTLD
+HOLDINGS, New-gTLD
+EQUIPMENT, New-gTLD
+xn--ngbc5azd, New-gTLD
+xn--80asehdb, New-gTLD
+xn--80aswg, New-gTLD
+xn--unup4y, New-gTLD);
\ No newline at end of file
diff --git a/conf/tlds.d/new_gtlds-20150614-20151223.csv b/conf/tlds.d/new_gtlds-20150614-20151223.csv
new file mode 100644
index 0000000..67b4c69
--- /dev/null
+++ b/conf/tlds.d/new_gtlds-20150614-20151223.csv
@@ -0,0 +1,208 @@
+# Format is: alabel, type
+
+MOBILY, New-gTLD
+xn--mgbb9fbpob, New-gTLD
+STAR, New-gTLD
+PID, New-gTLD
+CONTACT, New-gTLD
+WEBER, New-gTLD
+ANALYTICS, New-gTLD
+JMP, New-gTLD
+xn--jlq61u9w7b, New-gTLD
+SAS, New-gTLD
+FORD, New-gTLD
+STORAGE, New-gTLD
+LINCOLN, New-gTLD
+KFH, New-gTLD
+WANGGOU, New-gTLD
+SHELL, New-gTLD
+xn--pbt977c, New-gTLD
+xn--kpu716f, New-gTLD
+KPN, New-gTLD
+xn--ngbe9e0a, New-gTLD
+TRAVELERSINSURANCE, New-gTLD
+WATCHES, New-gTLD
+xn--eckvdtc9d, New-gTLD
+TUSHU, New-gTLD
+TRV, New-gTLD
+REDUMBRELLA, New-gTLD
+xn--mgbt3dhd, New-gTLD
+PARS, New-gTLD
+ZERO, New-gTLD
+SMILE, New-gTLD
+BOT, New-gTLD
+JOY, New-gTLD
+BUY, New-gTLD
+GOT, New-gTLD
+FAST, New-gTLD
+JOT, New-gTLD
+PIN, New-gTLD
+READ, New-gTLD
+CIRCLE, New-gTLD
+BOOK, New-gTLD
+SAFE, New-gTLD
+CALL, New-gTLD
+ROOM, New-gTLD
+LIKE, New-gTLD
+AUTHOR, New-gTLD
+TRAVELERS, New-gTLD
+NOWRUZ, New-gTLD
+FIRESTONE, New-gTLD
+TCI, New-gTLD
+SHARP, New-gTLD
+SALON, New-gTLD
+SHIA, New-gTLD
+INSURANCE, New-gTLD
+NORTON, New-gTLD
+MED, New-gTLD
+SYMANTEC, New-gTLD
+SFR, New-gTLD
+BOEHRINGER, New-gTLD
+BOSTIK, New-gTLD
+LAMBORGHINI, New-gTLD
+BUGATTI, New-gTLD
+AUDI, New-gTLD
+VIP, New-gTLD
+VERISIGN, New-gTLD
+BROADWAY, New-gTLD
+COMSEC, New-gTLD
+GRAINGER, New-gTLD
+TAB, New-gTLD
+FAIRWINDS, New-gTLD
+CITYEATS, New-gTLD
+LIFESTYLE, New-gTLD
+CREDITUNION, New-gTLD
+VANA, New-gTLD
+PLAYSTATION, New-gTLD
+FERRERO, New-gTLD
+ROCHER, New-gTLD
+APPLE, New-gTLD
+AARP, New-gTLD
+BEATS, New-gTLD
+PING, New-gTLD
+MEO, New-gTLD
+SAPO, New-gTLD
+SBS, New-gTLD
+GUCCI, New-gTLD
+LANDROVER, New-gTLD
+JAGUAR, New-gTLD
+RWE, New-gTLD
+ZARA, New-gTLD
+MUTUELLE, New-gTLD
+ARTE, New-gTLD
+ARAMCO, New-gTLD
+xn--mgba3a3ejt, New-gTLD
+DELL, New-gTLD
+CIPRIANI, New-gTLD
+KINDER, New-gTLD
+YAMAXUN, New-gTLD
+MOI, New-gTLD
+VIRGIN, New-gTLD
+MTR, New-gTLD
+CLUBMED, New-gTLD
+BOM, New-gTLD
+STOCKHOLM, New-gTLD
+KIA, New-gTLD
+FINAL, New-gTLD
+SEVEN, New-gTLD
+HYUNDAI, New-gTLD
+OBI, New-gTLD
+LTD, New-gTLD
+BMS, New-gTLD
+SECURITY, New-gTLD
+LINDE, New-gTLD
+STADA, New-gTLD
+THEATRE, New-gTLD
+PROTECTION, New-gTLD
+CAR, New-gTLD
+CSC, New-gTLD
+AMICA, New-gTLD
+STC, New-gTLD
+VIVA, New-gTLD
+STCGROUP, New-gTLD
+AAA, New-gTLD
+GEA, New-gTLD
+ACO, New-gTLD
+xn--efvy88h, New-gTLD
+MOM, New-gTLD
+FAMILY, New-gTLD
+SEEK, New-gTLD
+FAGE, New-gTLD
+CEB, New-gTLD
+GROUP, New-gTLD
+GIVING, New-gTLD
+XPERIA, New-gTLD
+BOOTS, New-gTLD
+CHANEL, New-gTLD
+WINE, New-gTLD
+VIN, New-gTLD
+LIXIL, New-gTLD
+xn--tckwe, New-gTLD
+xn--fhbei, New-gTLD
+xn--c2br7g, New-gTLD
+xn--9dbq2a, New-gTLD
+xn--11b4c3d, New-gTLD
+xn--3pxu8k, New-gTLD
+xn--t60b56a, New-gTLD
+xn--j1aef, New-gTLD
+xn--pssy2u, New-gTLD
+xn--42c2d9a, New-gTLD
+xn--mk1bu44c, New-gTLD
+IPIRANGA, New-gTLD
+LEXUS, New-gTLD
+PET, New-gTLD
+MAN, New-gTLD
+TOYOTA, New-gTLD
+SANOFI, New-gTLD
+TATAMOTORS, New-gTLD
+SRL, New-gTLD
+BET, New-gTLD
+ICE, New-gTLD
+ITAU, New-gTLD
+NOKIA, New-gTLD
+LANCASTER, New-gTLD
+UBS, New-gTLD
+ISTANBUL, New-gTLD
+DELTA, New-gTLD
+IST, New-gTLD
+HSBC, New-gTLD
+IINET, New-gTLD
+ORANGE, New-gTLD
+BENTLEY, New-gTLD
+GAME, New-gTLD
+JPRS, New-gTLD
+BCN, New-gTLD
+STUDIO, New-gTLD
+BARCELONA, New-gTLD
+AIRTEL, New-gTLD
+LIVE, New-gTLD
+APP, New-gTLD
+SAKURA, New-gTLD
+REALTY, New-gTLD
+FORUM, New-gTLD
+HOTELES, New-gTLD
+TELEFONICA, New-gTLD
+BNL, New-gTLD
+CLOUD, New-gTLD
+BRADESCO, New-gTLD
+MOVISTAR, New-gTLD
+OMEGA, New-gTLD
+SWATCH, New-gTLD
+LAW, New-gTLD
+SKYPE, New-gTLD
+SCOR, New-gTLD
+OFFICE, New-gTLD
+STARHUB, New-gTLD
+VISTA, New-gTLD
+COMMBANK, New-gTLD
+VISTAPRINT, New-gTLD
+RICOH, New-gTLD
+NETBANK, New-gTLD
+CBA, New-gTLD
+GENTING, New-gTLD
+DRIVE, New-gTLD
+PLAY, New-gTLD
+AEG, New-gTLD
+CROWN, New-gTLD
+STATOIL, New-gTLD
+BHARTI, New-gTLD
\ No newline at end of file
diff --git a/configure.ac b/configure.ac
new file mode 100755
index 0000000..19ca5e1
--- /dev/null
+++ b/configure.ac
@@ -0,0 +1,103 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+
+#                                               -*- Autoconf -*-
+# Process this file with autoconf to produce a configure script.
+
+AC_PREREQ(2.61)
+AC_INIT(Hedgehog, 2.4.2,  dns-stats-users@dns-stats.org)
+AC_CONFIG_SRCDIR([tools/hedgehogctl.in])
+AC_CONFIG_HEADER([config.h])
+
+AC_DEFINE_UNQUOTED([DB_SCHEMA_VERSION],[14],[Database version])
+AC_SUBST([DB_SCHEMA_VERSION],[14])
+AC_SUBST([VERSION_NAME],[Bare-bellied])
+
+AC_CONFIG_MACRO_DIR([m4])
+
+AM_INIT_AUTOMAKE([foreign])
+m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])
+
+# Checks for programs.
+AC_PROG_CC
+AC_PROG_CPP
+AC_PROG_CXX
+AC_PROG_AWK
+AC_PROG_SED
+AC_PROG_MAKE_SET
+AC_PROG_INSTALL
+AC_PROG_LN_S
+AC_PROG_LIBTOOL
+LT_INIT
+AM_PATH_PYTHON
+AC_PATH_PROG(HELP2MAN, help2man)
+if test -z "$HELP2MAN"
+then
+  AC_MSG_ERROR("help2man not found")
+fi
+
+AC_CONFIG_FILES(Makefile conf/Makefile tools/Makefile)
+
+WEB=1
+DM=1
+ACX_COMPONENTS
+AS_IF([test $WEB -eq 1],[AC_MSG_NOTICE([Building Web Component...])
+  AC_CONFIG_FILES(www/Makefile)
+
+],[AC_MSG_NOTICE([Not building Web Component...])])
+
+AS_IF([test $DM -eq 1],[AC_MSG_NOTICE([Building Data Manager Component...])
+
+  # Checks for header files.
+  AC_CHECK_HEADERS([arpa/inet.h fcntl.h netinet/in.h stdlib.h string.h sys/param.h unistd.h])
+
+  # Checks for typedefs, structures, and compiler characteristics.
+  AC_C_CONST
+  AC_STRUCT_TM
+  AC_TYPE_INT16_T
+  AC_TYPE_INT32_T
+  AC_TYPE_INT64_T
+  AC_TYPE_INT8_T
+  AC_TYPE_PID_T
+  AC_TYPE_SIZE_T
+  AC_TYPE_UINT16_T
+  AC_TYPE_UINT32_T
+  AC_TYPE_UINT64_T
+  AC_TYPE_UINT8_T
+  # Checks for library functions.
+  AC_FUNC_STRFTIME
+  AC_FUNC_MALLOC
+  AC_FUNC_MKTIME
+  AC_FUNC_REALLOC
+  AC_CHECK_FUNCS([gettimeofday mempcpy memset strerror])
+
+  BOOST_REQUIRE([1.41.0])
+  BOOST_SYSTEM
+  BOOST_FILESYSTEM
+  BOOST_PROGRAM_OPTIONS
+  BOOST_CONVERSION
+  BOOST_UNORDERED
+  BOOST_FOREACH
+  BOOST_STRING_ALGO
+  BOOST_TOKENIZER
+  
+  # Checks for libraries.
+  ACX_POSTGRES
+  ACX_LIBPQXX
+
+  CXXFLAGS="${CXXFLAGS} -Wall -pedantic \
+           -pedantic-errors -Wextra"
+
+  AC_CONFIG_FILES(src/Makefile)
+  
+],[AC_MSG_NOTICE([Not building Data Manager Component...])])
+
+AM_CONDITIONAL([DM], [test $DM -eq 1])
+AM_CONDITIONAL([WEB], [test $WEB -eq 1])
+
+AC_OUTPUT
diff --git a/docs/.DS_Store b/docs/.DS_Store
new file mode 100644
index 0000000..3e9de34
Binary files /dev/null and b/docs/.DS_Store differ
diff --git a/docs/Hedgehog_Installation_Guide_(from Packages).pdf b/docs/Hedgehog_Installation_Guide_(from Packages).pdf
new file mode 100644
index 0000000..b1c9dda
Binary files /dev/null and b/docs/Hedgehog_Installation_Guide_(from Packages).pdf differ
diff --git a/docs/Hedgehog_Tools_Guide.pdf b/docs/Hedgehog_Tools_Guide.pdf
new file mode 100644
index 0000000..ddd117c
Binary files /dev/null and b/docs/Hedgehog_Tools_Guide.pdf differ
diff --git a/docs/Hedgehog_Upgrade_Guide_(to_Packages).pdf b/docs/Hedgehog_Upgrade_Guide_(to_Packages).pdf
new file mode 100644
index 0000000..356f677
--- /dev/null
+++ b/docs/Hedgehog_Upgrade_Guide_(to_Packages).pdf
@@ -0,0 +1,49 @@
+%PDF-1.4
+%âãÏÓ
+4 0 obj
+<</Filter/FlateDecode/Length 301>>stream
+xËNÃ0E÷ó³qb§IX"ñ"#'ÒÄ´¿Ã§bG}JHÈ²4wlß9WîáNÇLF,ËP¸W°8ã"Mp1>ûóD_àã£ap¿D¦,>êõAËqé¥¼(·jxÎÛåªpóa³U	ÑÜP¦-ª®ÈTTÛ
+ß6ÓðqjÝbÌ$p´s_«U1{Æ²<¿ðôB0Ìªn4¶ZêF44®ù¢k»~Ó~ÆÜÎ¶èhMz ÓØóF¶Ä#è¶Ñ¸ÑÅ·®h`?çd>­ø+-ç3°g{=wý7Ô"ÍUMÎ3ûÝÙ.ÆÆví§ÆÁÒ:Cþ}J¶~	¿dþø
+endstream
+endobj
+1 0 obj
+<</Contents 4 0 R/Type/Page/Resources<</ProcSet [/PDF /Text /ImageB /ImageC /ImageI]/Font<</F1 2 0 R/F2 3 0 R>>>>/Parent 5 0 R/MediaBox[0 0 595.28 841.88]>>
+endobj
+7 0 obj
+<</Dest[1 0 R/XYZ 0 793.88 0]/Title(Hedgehog Upgrade Guide: 2.4.1 to 2.4.2)/Parent 6 0 R>>
+endobj
+6 0 obj
+<</Type/Outlines/Count 1/First 7 0 R/Last 7 0 R>>
+endobj
+2 0 obj
+<</Subtype/Type1/Type/Font/BaseFont/Helvetica-Bold/Encoding/WinAnsiEncoding>>
+endobj
+3 0 obj
+<</Subtype/Type1/Type/Font/BaseFont/Helvetica/Encoding/WinAnsiEncoding>>
+endobj
+5 0 obj
+<</Kids[1 0 R]/Type/Pages/Count 1/ITXT(2.1.7)>>
+endobj
+8 0 obj
+<</PageMode/UseOutlines/Type/Catalog/Outlines 6 0 R/Pages 5 0 R>>
+endobj
+9 0 obj
+<</ModDate(D:20190108144708Z)/CreationDate(D:20190108144708Z)/Producer(iText 2.1.7 by 1T3XT)>>
+endobj
+xref
+0 10
+0000000000 65535 f 
+0000000383 00000 n 
+0000000726 00000 n 
+0000000819 00000 n 
+0000000015 00000 n 
+0000000907 00000 n 
+0000000661 00000 n 
+0000000555 00000 n 
+0000000970 00000 n 
+0000001051 00000 n 
+trailer
+<</Info 9 0 R/ID [<a7d465f00f59a80196f86ed6bf292470><d65fc21bb7a817f5f06274abf2f3158e>]/Root 8 0 R/Size 10>>
+startxref
+1161
+%%EOF
diff --git a/docs/Hedgehog_User_Guide.pdf b/docs/Hedgehog_User_Guide.pdf
new file mode 100644
index 0000000..a93730d
Binary files /dev/null and b/docs/Hedgehog_User_Guide.pdf differ
diff --git a/docs/src/Hedgehog_Installation_Guide_(from_Packages).xml b/docs/src/Hedgehog_Installation_Guide_(from_Packages).xml
new file mode 100644
index 0000000..6fac47a
--- /dev/null
+++ b/docs/src/Hedgehog_Installation_Guide_(from_Packages).xml
@@ -0,0 +1,347 @@
+
+<p><ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="f9e02cbb-e6f6-4fe6-b9b0-812aded99f27" /></p>
+<p>The following instructions describe how to install Hedgehog from Ubuntu Packages on Ubuntu 14.04 Server and set up imports of data.</p>
+<p><br /></p><ac:structured-macro ac:name="numberedheadings" ac:schema-version="1" ac:macro-id="b97e1073-8939-41e4-9abc-f56b74e227db"><ac:rich-text-body>
+<h1>Requirements</h1>
+<h2>Platform support</h2>
+<p>Hedgehog is currently only supported on Ubuntu 14.04 LTS Server.</p>
+<h2>Database support</h2>
+<p>Hedgehog has been tested with PostgreSQL 9.3.X and 9.6.X. The installation of the database is not covered in detail since it may or may not be co-located with the other hedgehog components.</p>
+<p>The <span>hedgehog-database package (see below) </span>will suggest the following two packages:</p>
+<ul>
+<li>
+<p>postgresql-9.3-ip4r</p></li>
+<li>
+<p>postgresql-plpython-9.3</p></li></ul>
+<p>Versions of these packages appropriate to the version of PostgreSQL installed are required.</p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="d352c3b5-d0f4-4f56-93a5-e788432f549f"><ac:rich-text-body>
+<p>The Hedgehog servers and PostgreSQL must both be configured to use UTC!</p></ac:rich-text-body></ac:structured-macro>
+<h2>General</h2>
+<h3>Home directory permissions</h3>
+<p>Some users set the permissions on their home directory such that other users cannot read or execute that directory. In this case when running scripts that update the database as the <em>hedgehog</em> or <em>postgres</em> (user via 'sudo -u &lt;user&gt;') meaningless <span>messages are generated from psql about being unable to cd into logged in users home directory. To avoid this</span> spurious output change directory to one that allows other users to read/execute (e.g. /tmp) and run the scripts from there.</p>
+<h1>Hedgehog Installation</h1>
+<h2>Hedgehog packages</h2>
+<p>The Hedgehog PPA is located at</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="3aec0f80-ac4a-4605-8897-f99062b8fcef"><ac:plain-text-body><![CDATA[sudo add-apt-repository ppa:dns-stats/presenter]]></ac:plain-text-body></ac:structured-macro>
+<p>Hedgehog comprises 3 main components which can be run on the same or different machines. Configure the additional PPA's as required and then install the main packages from the list below:</p>
+<table class="wrapped"><colgroup><col /><col /><col /><col /><col /></colgroup>
+<tbody>
+<tr>
+<th>Component</th>
+<th>Description</th>
+<th><span>Main Package</span></th>
+<th colspan="1">Optional Package</th>
+<th colspan="1">Additional Required PPAs</th></tr>
+<tr>
+<td colspan="1">Database</td>
+<td colspan="1">PostgreSQL database for Hedgehog</td>
+<td colspan="1">hedgehog-database</td>
+<td colspan="1"><br /></td>
+<td colspan="1">
+<p>sudo add-apt-repository ppa:ondrej/pkg-nlnetlabs</p></td></tr>
+<tr>
+<td rowspan="2"><span>Data Manager</span></td>
+<td>
+<p>Scripts for database population and management (including rssacd)</p>
+<p>Scripts for XML/DAT processing</p></td>
+<td><span>hedgehog-data-manager</span></td>
+<td colspan="1"><br /></td>
+<td colspan="1"><br /></td></tr>
+<tr>
+<td colspan="1">(Optional) Apache config for webdav based XML uploads</td>
+<td colspan="1"><br /></td>
+<td colspan="1"><span>hedgehog-webdav-upload</span></td>
+<td colspan="1"><br /></td></tr>
+<tr>
+<td>Web GUI</td>
+<td>Web GUI front-end</td>
+<td><span>hedgehog-gui</span></td>
+<td colspan="1"><br /></td>
+<td colspan="1">
+<p>sudo add-apt-repository ppa:opencpu/rapache</p></td></tr></tbody></table>
+<p>Notes:</p>
+<ul>
+<li>Multiple Web GUI's can be connected to the same database.</li>
+<li>The upload mechanism for XML/DAT files is for the user to decide (ssh, rsync, Apache, etc.). A package for upload using Apache is provided for convenience.</li></ul>
+<p>For reference the other Hedgehog packages are:</p>
+<table class="wrapped">
+<tbody>
+<tr>
+<th>Description</th>
+<th>Required by</th>
+<th><span>Package</span></th></tr>
+<tr>
+<td>Common package</td>
+<td><span>All main hedgehog packages</span></td>
+<td>hedgehog-common</td></tr>
+<tr>
+<td colspan="1">RPostgresHelper</td>
+<td colspan="1">Web Gui</td>
+<td colspan="1">r-hedgehog-rpostgresqlhelper</td></tr>
+<tr>
+<td colspan="1">Documentation</td>
+<td colspan="1"><br /></td>
+<td colspan="1">hedgehog-doc</td></tr></tbody></table>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="30c0ff9d-f3ef-44b1-9d42-b14508dddb31"><ac:rich-text-body>
+<p><span>Two system users are used by Hedgehog, which are created by the packages:</span></p>
+<table class="wrapped">
+<tbody>
+<tr>
+<th>User</th>
+<th>Default</th>
+<th colspan="1">Component</th>
+<th>Note</th>
+<th colspan="1">Created by</th></tr>
+<tr>
+<td>Database owner</td>
+<td>hedgehog</td>
+<td colspan="1">
+<p>Data Manager</p></td>
+<td>
+<p>This is the user that will own the database created by</p>
+<p>Hedgehog and the top level datafile directories.</p></td>
+<td colspan="1">hedgehog-common package</td></tr>
+<tr>
+<td>Database read user</td>
+<td>www-data</td>
+<td colspan="1">
+<p>Web front-end</p></td>
+<td>
+<p>This defaults to the same as the default apache user</p></td>
+<td colspan="1">apache2 package installation</td></tr></tbody></table>
+<p><span><br /></span></p></ac:rich-text-body></ac:structured-macro>
+<p><br /></p>
+<h2>Web GUI only - Manual install of R packages</h2>
+<p>These instructions&nbsp;are for R packages that must be installed using R's built-in compilation tools (there is no Ubuntu package for them).</p>
+<p>To install interactively: (Some can repositories don't contain packages for all versions of R so if this fails quit R and re-try a different repo)</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="41d72df2-2e8d-49a1-b22e-ea5313c97f16"><ac:plain-text-body><![CDATA[sudo R
+install.packages(c("brew","Cairo","googleVis","R.utils","yaml","plyr"))
+q()
+# If you are prompted to save workspace image y/n/c, choose no. ]]></ac:plain-text-body></ac:structured-macro>
+<p><span>For scripting purposes a repo can be specified by using a command of the form:</span></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="e85ac8e9-5693-47be-bf61-a712d38cd191"><ac:plain-text-body><![CDATA[install.packages("name", repos='http://cran.rstudio.com/')]]></ac:plain-text-body></ac:structured-macro>
+<p><span>Hedgehog is tested against version 3.1.1 of R.&nbsp;</span>Hedgehog requires at least the versions below of each package.</p>
+<table class="wrapped">
+<thead>
+<tr>
+<th>
+<div class="tablesorter-header-inner">R Package</div></th>
+<th>
+<div class="tablesorter-header-inner">Supported Version</div></th></tr></thead>
+<tbody>
+<tr>
+<td>brew</td>
+<td>1.0-6</td></tr>
+<tr>
+<td>Cairo</td>
+<td>1.5-9</td></tr>
+<tr>
+<td>googleVis</td>
+<td>
+<p>0.5.10</p></td></tr>
+<tr>
+<td colspan="1">R.utils</td>
+<td colspan="1">2.2.0</td></tr>
+<tr>
+<td colspan="1">yaml</td>
+<td colspan="1">2.1.13</td></tr>
+<tr>
+<td colspan="1">plyr</td>
+<td colspan="1">
+<p class="p1"><span class="s1">1.8.3</span></p></td></tr></tbody></table>
+<p><br /></p>
+<h1>Hedgehog configuration</h1>
+<ul>
+<li>For the Web GUI and Data Manager components, ensure the /etc/hedgehog/hedgehog.yaml configuration file contains usernames and passwords that will match those configured in the database (see next section).</li>
+<li>Also configure the database parameters (host, port and name) as required.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="b90d6358-dbe9-4d83-bfe3-f3addfa11f8e"><ac:plain-text-body><![CDATA[database: 
+  host         : /var/run/postgresql # specify a host for the postgresql DB. If
+                                     # this begins with a slash, it specifies the
+                                     # directory in which the socket file is 
+                                     # stored.
+  port         : 5432      # specify port for the postgresql DB.
+  name         : hedgehog  # specify dbname for the postgresql DB.
+  owner        : hedgehog  # specify a user to own the postgresql DB. 
+                           # [Required for Data Manager component]
+  owner_pass   :           # specify a password for the owner user if needed.
+  read_user    : www-data  # specify a read user for the postgresql DB. 
+                           # [Required for Web front-end component]
+  read_pass    :           # specify a password for the read user if needed.
+]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="67324d78-e7f4-4511-9743-e5f797434297"><ac:rich-text-body>
+<p><span>Do not edit the 'directories' section of this file as it is auto-generated.</span></p></ac:rich-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="5be45376-3b92-49ac-b912-64b943762b7e"><ac:rich-text-body>
+<p>Depending on you PostgreSQL configuration you may need to add passwords to this file. If you do then be aware that, by default, this file is readable by all users. You should restrict access to just the hedgehog user on a Data Manager only machine or the hedgehog and www-data user on a combined Data Manager and Web GUI.</p></ac:rich-text-body></ac:structured-macro>
+<p><br /></p>
+<h2>Database configuration</h2>
+<h3>Create a database</h3>
+<p>Ask your DBA to create the necessary database. This is a script to help them. It creates the DATABASE, SCHEMA, FUNCTIONS, EXTENSIONS, LANGUAGES, USERS and ROLES needed to run hedgehog (using default values), and can optionally accept a user specified database name and read/write user names.</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="16c3ab93-2380-4a94-8c94-8a9a71249823"><ac:plain-text-body><![CDATA[sudo -u postgres /usr/bin/hedgehog_database_create
+ 
+#If you want to use passwords do something like this or use peer authentication
+sudo -u postgres psql
+alter user hedgehog password 'hedgehog123';
+alter user "www-data" password 'www123';]]></ac:plain-text-body></ac:structured-macro>
+<p>Modify the PostgreSQL configuration as so:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="e4620eed-e360-454c-9590-b130d1ca55e8"><ac:plain-text-body><![CDATA[sudo vi /etc/postgresql/9.3/main/postgresql.conf
+# uncomment and set 'extra_float_digits = 1'
+sudo service postgresql reload]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="8916bafd-dbfe-4999-ba7c-b5a4a4417c2b"><ac:rich-text-body>
+<p>We recommend that the 'pgtune' tool is used to obtain an optimal configuration for PostgreSQL. For example:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="48a4dfb3-f0c2-4f13-9942-c1a7c69069d8"><ac:plain-text-body><![CDATA[pgtune -i /etc/postgresql/9.3/main/postgresql.conf -c 200 -T DW]]></ac:plain-text-body></ac:structured-macro>
+<p>Also some queries to the database trigger DNS lookups from functions in the database. It may be optimal to run a local resolver such as Unbound.</p></ac:rich-text-body></ac:structured-macro>
+<p><br /></p>
+<h2><span>Data Manager configuration</span></h2>
+<h3>Specify the nodes/servers</h3><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="33b7d9dd-5c22-40dd-b1cb-2cc833ff1b7a"><ac:rich-text-body>
+<p><span>For this version of Hedgehog the servers and nodes to be processed and displayed must be specified manually as described here.</span></p></ac:rich-text-body></ac:structured-macro>
+<ul>
+<li>Edit the the&nbsp;<em>/etc/hedgehog/nodes.csv</em>&nbsp;file to specify the servers, nodes and grouping to be used (example format is provided with entries commented out).</li>
+<li>Note that the current GUI layout is optimised for nodes with short names (&lt;6 characters) of the same length</li></ul>
+<h3>Specify the server service addresses (if needed)</h3>
+<p><span>If you want to restrict the server IP addresses that are displayed in the 'By server IP address table' then edit the <em>/etc/hedgehog/service_addr.csv</em> file to add the list of addresses to be displayed.</span></p>
+<h3><span>Directory permissions</span></h3>
+<p><span>The </span>/var/lib/hedgehog/data/ is used to store incoming XML files</p><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="c988fd5b-f033-4348-a2e6-38dcd2795b97"><ac:rich-text-body>
+<p>You may also need to alter the permissions on this directory to allow uploads via your chosen mechanism</p></ac:rich-text-body></ac:structured-macro>
+<h3>Create the database tables</h3>
+<p>Run the command below noting the following:</p>
+<ul>
+<li>If you have historic data to import then use the -m flag to specify the month of the oldest data that will need importing. Otherwise the database tables will be created to hold data from this month onwards.</li>
+<li>Note that this script will also create the directory structure for all the specified servers and nodes under the <em>data</em> directory if it does not exist</li>
+<li>(Note the insertion of the GeoIP data can take some time)</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="fe8954a6-8071-4c6c-917f-f2af4cfc7e64"><ac:plain-text-body><![CDATA[sudo -u hedgehog /usr/bin/hedgehogctl database_init]]></ac:plain-text-body></ac:structured-macro>
+<h3>(Optional) Webdav upload</h3>
+<p>If using webdav to upload XML files then add the following to the /etc/apache2/envvars file:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="38cd9a9c-70a2-4865-9c72-811dda53e3d2"><ac:plain-text-body><![CDATA[umask 002]]></ac:plain-text-body></ac:structured-macro>
+<p>Then enable the dependancies:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="3bcca21b-ddef-46de-9f3e-d4f4d4d5edbd"><ac:plain-text-body><![CDATA[sudo a2enmod dav
+sudo a2enmod ssl
+suod a2enmod dav_fs]]></ac:plain-text-body></ac:structured-macro>
+<p>And then enable the hedgehog webdav site:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="5136fdd1-28a2-4901-a344-a7702383709d"><ac:plain-text-body><![CDATA[sudo a2ensite hedgehog-webdav-upload]]></ac:plain-text-body></ac:structured-macro>
+<p><br /></p>
+<h2>Web GUI configuration</h2>
+<p>Check the parameters in the/etc/hedgehog/hedgehog_gui.yaml file, which specifies parameters controlling the behaviour of the web front end. See the &quot;Plot Caching&quot; section in the user guide for a more detailed description of when plots are cached.</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="dd531aa2-f9a3-4f1d-a917-9ca914ec4530"><ac:plain-text-body><![CDATA[--- 
+# YAML config for hedgehog GUI.
+# NOTE: If this file is changed then apache must be restarted for the changes to take effect
+www:
+  default_plot_type             : interactive # 'static'      -> png plots
+                                              # 'interactive' -> googlevis plots
+  default_interactive_plot_type : svg         # 'flash' -> plot requires flash
+                                              # 'svg'   -> plot is SVG/VML and 
+                                              # does not require flash (but with svg
+                                              # plots some legends do not wrap properly)
+  default_node_grouping         : instance    # choose from 'none', 'instance', 'city' or
+                                              # 'country'
+  use_plot_caching              : 1           # '1' -> true, use cached plots when possible
+                                              # '0' -> false, never use cached plots
+  caching_delay_in_hours        : 1           # If 'use_plot_caching=1' then only plots with
+                                              # an end time earlier than this number of 
+                                              # hours ago are cached. More recent plots are
+                                              # not cached as data may still be importing
+  presentation_delay_in_hours   : 0           # Number of hours behind now for which the
+                                              # GUI will display data
+  support_url                   :             # configurable target for "Support" external
+                                              # link on Homepage. The default dns-stats.org
+                                              # issue tracker used if left blank
+  default_server                :             # Optionally specify the default server to
+                                              # use in the server drop-down 
+                                              # (default is first alphabetically)
+  default_plot_name             : by_region   # default plot name (See Hedgehog internal 
+                                                plot name in the user guide)
+  enable_node_selection         : 1           # '1' -> true, Show the node radio button
+                                              # '0' -> false, hide the node radio button]]></ac:plain-text-body></ac:structured-macro>
+<h3>Apache configuration</h3>
+<p>Depending on your exact installation choices and apache configuration y<span style="line-height: 1.4285715;">ou may need to disable the default site using the following command:</span></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="98d7d7c3-f17d-4110-985b-113256902536"><ac:plain-text-body><![CDATA[sudo a2dissite 000-default.conf]]></ac:plain-text-body></ac:structured-macro>
+<p><br /></p>
+<ul>
+<li>Add the Hedgehog configuration files to apache and enable the site (this file name can be changed if required to match any local apache policy):</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="01f53792-d09e-4d76-886f-7cf5b123ac37"><ac:plain-text-body><![CDATA[sudo a2ensite hedgehog.conf]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="be50c57a-e595-474b-a476-1449dc6766a8"><ac:rich-text-body>
+<p><span>apache/rapache write some of their logs to user.* so it can be useful to change the syslog config: </span></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="d573ab6b-d29e-44e4-8d87-41fe0a8c74ea"><ac:plain-text-body><![CDATA[sudo vi /etc/rsyslog.d/50-default.conf]]></ac:plain-text-body></ac:structured-macro>
+<p>Uncomment the line&nbsp;beginning&nbsp;'user.*'.&nbsp;</p></ac:rich-text-body></ac:structured-macro>
+<p><br /></p>
+<ul>
+<li>Finally, reload apache:</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="744ab3cb-e4f9-43f0-8174-db3f3eda513e"><ac:plain-text-body><![CDATA[sudo service apache2 reload]]></ac:plain-text-body></ac:structured-macro>
+<p><br /></p>
+<p>At this point you should test that you can see the servers and nodes in the web front end at the URL http://&lt;server-name&gt;/hedgehog</p>
+<h1>Importing data</h1>
+<p>Hedgehog can process data in the following 3 ways:</p>
+<table class="wrapped">
+<tbody>
+<tr>
+<th>Source format</th>
+<th>Output format</th>
+<th><br /></th></tr>
+<tr>
+<td>XML</td>
+<td>Database</td>
+<td>For real time uploads</td></tr>
+<tr>
+<td colspan="1">DAT</td>
+<td colspan="1">Database</td>
+<td colspan="1">For import of historic data</td></tr>
+<tr>
+<td colspan="1">XML</td>
+<td colspan="1">DAT</td>
+<td colspan="1">For backwards compatibility with DSC</td></tr></tbody></table>
+<p><br /></p>
+<p>In each case the <em>/usr/bin/refile_and_grok</em> script is used, it is simply given different parameters:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="a0503194-2b02-4ad3-8826-e8c87d2c93d7"><ac:plain-text-body><![CDATA[> refile_and_grok -h
+
+refile_and_grok - finds all input files in working directory and processes to output format
+
+
+-w Working directory to search for input files (default: )
+-i Input file format <XML|DAT> (default: XML)
+-o Output file format <DAT|DB> (default: DB)
+-c Non-interactive mode - use this flag when being run by a cron job
+-s Start date from which to process incoming data (XML input only)
+-r Disable processing of rssac data. Default is to process all data.
+-R Reserved processors. Number of CPUS processors to exclude from import (default 0).
+-a Append output to the refile_and_grok.stdout file (default is overwrite)
+-h Show this help.]]></ac:plain-text-body></ac:structured-macro>
+<p><br /></p>
+<h2>Importing historical .DAT data</h2><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="8937cd73-d9f5-4af6-b41d-56e9ef9b925c"><ac:plain-text-body><![CDATA[sudo -u <DB_OWNER> /usr/bin/refile_and_grok -i DAT]]></ac:plain-text-body></ac:structured-macro>
+<p>Be aware that this can take a long time if there is a significant amount of historic data and it may be advisable to run this in stages.</p>
+<h2>Importing real-time .XML data</h2>
+<h3>Manually</h3>
+<ul>
+<li>This can be done manually by running the <em>refile_and_grok</em> script (consider running this nohup as it may take a while depending on how much data there is to process).</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="71957016-03d6-4d53-98b6-847f9b2e0ddb"><ac:plain-text-body><![CDATA[sudo -u <DB_OWNER> /usr/bin/refile_and_grok]]></ac:plain-text-body></ac:structured-macro>
+<ul>
+<li>A snapshot of the progress of the data import can be generated by running the command below:</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="51ca4ae5-494c-47f2-aacd-ca30d38c6019"><ac:plain-text-body><![CDATA[sudo -u <DB_OWNER> /usr/bin/hedgehogctl datafiles_create_summary]]></ac:plain-text-body></ac:structured-macro>
+<h3>Automatically</h3>
+<ul>
+<li>Configure a regular cron job for <em>refile_and_grok</em> as shown below</li></ul>
+<h2>Importing zone-size and load-time data for RSSAC</h2>
+<p>To do this run&nbsp;the&nbsp;<em>rssacd</em>&nbsp;demon specifying the server of interest. This listens for NOTIFY messages and after receiving one:</p>
+<ul>
+<li>obtains the zone size from the notifying server by performing an XFR and&nbsp;</li>
+<li>probes each node listed for the configured server&nbsp;with an IP address&nbsp;to calculate the zone load time&nbsp;</li></ul><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="ff2b3e0a-8fe8-406f-8c30-b0b73bbf70d1"><ac:rich-text-body>
+<p><span>This requires that management IP addresses are configured for the nodes via the nodes.csv file (also note that&nbsp;</span><em>rssacd</em><span>&nbsp;needs restarting if new nodes are added).</span></p></ac:rich-text-body></ac:structured-macro>
+<p><em style="line-height: 1.42857;">rssacd</em>&nbsp;can be run from the command line for testing, or using an init script.</p>
+<p>From the command line:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="c5b57e0c-4417-415f-9f13-35848a6a5ec8"><ac:plain-text-body><![CDATA[/usr/sbin/rssacd --log ~/var/log/hedgehog/rssacd.log -s <server-name> -z <fully_qualified_zone_name>]]></ac:plain-text-body></ac:structured-macro>
+<p>From an init script:</p>
+<ul>
+<li>Configure the server name, TSIG key, etc. in the /etc/hedgehog/rssac.conf file</li></ul>
+<p>Then run:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="fad2282a-24f0-44d9-98bf-e718ff93ea98"><ac:plain-text-body><![CDATA[update-rc.d rssacd defaults]]></ac:plain-text-body></ac:structured-macro>
+<p><br /></p>
+<h1>Cron jobs</h1>
+<p>Several cron jobs need to be configured.</p>
+<h2>Data Manager component</h2>
+<p>Below is an example crontab for a typical data manager install (<em>sudo -u &lt;DB_OWNER&gt; crontab -e</em>).</p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="f24fdc6c-30e0-4911-ae3b-aef0fdccb1e9"><ac:rich-text-body>
+<p><strong><span>Note that the </span><em>database_manage_partitions </em><span>script </span><strong>MUST </strong> be run at least once a month to create the tables for next month or the import will fail.</strong></p></ac:rich-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="b841f66d-ab0d-49cf-8c1d-0d7178f85e3d"><ac:plain-text-body><![CDATA[# REQUIRED: 
+# Import XML data every 15 mins
+00,15,30,45 * * * * /usr/bin/refile_and_grok -c >> /var/log/hedgehog/refile_and_grok_xml_to_db.log 2>&1
+# Twice monthly job to make sure the DB tables for next month are created 
+# ahead of time
+0 6 15,28 * * /usr/bin/hedgehogctl database_manage_partitions >> /var/log/hedgehog/database_manage_partitions.log 2>&1
+
+# OPTIONAL:
+# Daily jobs to process RSSAC data. By default data is processed 
+# for a single day 1 week ago. Must be run before the rssac_generate_reports script (see Web GUI Component cron jobs)
+0 1 * * * /usr/bin/hedgehogctl database_process_rssac_data -D >> /var/log/hedgehog/database_process_rssac_data.log 2>&1
+# Weekly job to update the delegated TLDs from the IANA database.
+0 2 * * 0 <prefix>/bin/hedgehogctl database_update_tlds_from_zone >> /var/log/hedgehog/database_update_tlds_from_zone.log 2>&1
+# Monthly job to update the geoIP database.
+0 3 1 * * <prefix>/bin/hedgehogctl database_update_geoip >> /var/log/hedgehog/database_update_geoip.log 2>&1
+# Monthly job to remove empty xml directories that are older than 7 days old
+0 2 1 * * /usr/bin/hedgehogctl datafiles_rm_empty_xml_dirs -D >> /var/log/hedgehog/datafiles_rm_empty_xml_dirs.log 2>&1
+# Monthly job to tar up processed xml directories
+0 2 7 * * /usr/bin/hedgehogctl datafiles_tar_old_xml -D >> /var/log/hedgehog/datafiles_tar_old_xml.log 2>&1]]></ac:plain-text-body></ac:structured-macro>
+<h2>Web GUI component</h2>
+<p>Below is an example crontab for a typical web front-end install (<em>sudo -u &lt;DB_READ_USER&gt; crontab -e</em>)</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="760d5b53-0421-4e33-87d2-61beff00d451"><ac:plain-text-body><![CDATA[# OPTIONAL: 
+# Daily job to create cached plots for the previous day to make loading common plots  
+# quicker. Run a few hours after midnight so all data is uploaded.
+0 4 * * * /usr/bin/hedgehogctl plotcache_generate_cached_plots -D >> /var/log/hedgehog/plotcache_generate_cached_plots.log -D 2>&1
+# Daily job to generate RSSAC reports. By default report is generated 
+# for a single day 1 week ago. Must be run after the database_process_rssac_data script (see Data Manager cron jobs)
+0 3 * * * /usr/bin/hedgehogctl rssac_generate_reports >> /var/log/hedgehog/rssac_generate_reports.log 2>&1]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p></ac:rich-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p>
\ No newline at end of file
diff --git a/docs/src/Hedgehog_Tools_Guide.xml b/docs/src/Hedgehog_Tools_Guide.xml
new file mode 100644
index 0000000..e3f00aa
--- /dev/null
+++ b/docs/src/Hedgehog_Tools_Guide.xml
@@ -0,0 +1,211 @@
+
+<p>The Hedgehog tools are a set of scripts that are used to perform various tasks in terms of creating and maintaining the database and hedgehog files on disk. They can be run directly from <span>/libexec/hedgehog or</span>&nbsp;for convenience a wrapper script (<span>/bin/hedgehogctl</span>) is provided:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="b3c2535a-74e2-4382-956a-66431f19a058"><ac:plain-text-body><![CDATA[>hedgehogctl help
+ 
+Run a Hedgehog command
+
+Usage: hedgehogctl [help|list] [COMMAND] [command_options]]]></ac:plain-text-body></ac:structured-macro>
+<p>The available hedgehogctl commands are described below</p>
+<p><ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="8fff6759-8a52-4a19-929b-bc3886a8c215" /></p>
+<h2>Data Manager</h2>
+<p>These scripts should be run as the DB_OWNER user (<em>hedgehog</em> by default).</p>
+<h3>Database</h3>
+<h4>database_init</h4>
+<ul>
+<li>The script should be run after the database has been created (see the &lt;prefix&gt;/libexec/database_create helper script)</li>
+<li>This script populates the database with the plot and server/node data</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="35c6307d-3935-4e00-bffb-e37aa1ac19b6"><ac:plain-text-body><![CDATA[Create and populate the database tables with plot options and nodes.
+
+This calls the hedeghog_database_update_nodes.sh script which will populate the DB
+with the specified nodes and create a directory structure for them in the working
+data directory.
+
+Usage: database_init options
+
+Supported options:
+
+  -m Start month of oldest data to be imported 
+     (default is current month, format YYYY-MM)
+  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h4>database_manage_partitions</h4>
+<ul>
+<li>
+<p>This script creates new tables in the database to hold the imported data.&nbsp;</p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="02fd2867-ebc4-40f4-ac80-9a3515fc90a6"><ac:rich-text-body>
+<p><strong>It must be run at least once a month to create the tables for next month or the import will fail. It is recommended to configure a cron job as described in the&nbsp;Cron jobs section of the Installation guide.&nbsp;</strong></p></ac:rich-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p></li></ul>
+<ul>
+<li>Otherwise, the user will only have to run this script when new servers are added to an existing system.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="d5d85645-ac8c-4f52-a28d-93142894bb06"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Create a new set of partitioned data tables in the database to hold imported data.
+The partitions are created per month of data.
+ 
+Usage: database_manage_partitions options
+
+Supported options:
+    -m Month to create partitions for 
+       (default is next month, format YYYY-MM)
+    -h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h4>database_process_rssac_data</h4>
+<ul>
+<li>This script should be run to process the data ready for the <em>rssac_process_data </em>script.&nbsp;</li>
+<li>The importer inserts data from the XML files into the 'unique_source_raw' table. For each node, this script creates entries in the 'unique_source_summary' table based on the raw data and then (optionally) deletes the original raw data from the&nbsp;'unique_source_raw' table.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="1c107461-c555-46a2-aed9-0eb3d0823078"><ac:plain-text-body><![CDATA[Process RSSAC data for a specified 24 hour period
+ 
+Usage: database_process_rssac_data options
+
+Supported options:
+  -d Date on which to process RSSAC data 
+    (default is 7 days ago, format: YYYY-MM-DD)
+  -f Force overwrite of existing summary data
+  -D Also delete the raw unique_source data from the database to save space 
+    (default: no delete)
+  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h4>database_rm_old_rssac_data</h4>
+<ul>
+<li>For some systems, the unique source raw data can be unmanageably large. This script can be used to truncate the database tables to recover the disk space, for example it can be run once a month to remove all the raw data for the previous month</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="af1ff305-093e-450e-88c7-fd6c5cbc7f1b"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Delete all the raw unique_source data in the database for a given month. 
+All data older than this is also deleted.
+This script is intended to be run monthly to free disk space on systems 
+where the raw unique sources data becomes unmanagably large.
+
+Usage: database_rm_old_rssac_data options
+
+Supported options:
+  -m Month from which to delete all raw unique source data 
+       (default 2014-07: format YYYY-MM)
+       (default is 2 months ago when in the first half of the current month and
+        1 month ago when in the second half of the current month)
+  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h4>database_update_geoip</h4>
+<ul>
+<li>This script downloads a reference database of locations for use by the GEO maps. It should be periodically run to update with new location information.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="660d7d6b-270c-41fa-8cba-cd4f95aeb632"><ac:plain-text-body><![CDATA[Update the GeopIP list in the database. This reads input from /tmp
+Usage: database_update_geoip options
+Supported options:
+  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
+<h4>database_update_nodes</h4>
+<ul>
+<li>Run this script if new nodes or servers are added to the system</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4d9265e8-3cd1-4f78-b752-2861b5671147"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Add/update the servers and nodes in the database. This reads input from file
+called nodes.csv in the /usr/local/etc/hedgehog directory.
+ - An example nodes.csv file is installed if one does not exist.
+ - See the comments in that file for details of the format.
+ - No action is taken on servers/nodes that are in the database but are not in the 
+   input files.
+
+Usage: /usr/local/bin/hedgehog_database_update_nodes options
+
+Supported options:
+  -r rename/remove flag     This enables processing of the rename_or_remove field in nodes.csv
+                            If this flag is not given any lines in the input file that have
+                            text in the rename_or_remove field are ignored.
+                            Renaming a node will update the database and move the data
+                            directory to the new name.
+                            Removing a node will delete all associated data in the database
+                            but leave any data on disk untouched (for manual removal).
+                            Removing nodes could take a long time if you have a lot of data.
+  -d debug mode (set -x).
+  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
+<h4>database_update_service_addr</h4>
+<ul>
+<li>Run this script to control which server addresses are displayed in the 'By server IP address' plot. This can be useful to filter to just the server service addresses since DSC does not have such a fitler.&nbsp;</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="e88fc06e-c141-4890-89cb-321223aa31d4"><ac:plain-text-body><![CDATA[Update the Server service addreses list in the database.
+
+This list is only used to filter which addresses are displayed in the
+'By server IP address' plot. If there are any addresses in the list
+then only these addresses are displayed. If there are no entries 
+in the list then all the addresses for which there is data are displayed.
+By default this script reads the /opt/hedgehog/etc/hedgehog/service_addr.csv file and adds
+the addresses found there.
+After running this script it is advisable to remove any cached plots
+for the 'By server IP address' plot by using the plotcache_rm_cached_plots
+script.
+
+Usage: database_update_service_addr options
+
+Supported options:
+  -d Delete all existing entries in the server service address list
+  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
+<p><br /></p>
+<h4>database_update_tlds_from_zone</h4>
+<ul>
+<li>Run this script to update the categorisation of TLDs in the database from the root zone held in the IANA database.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="7ef95ada-9bd4-46b3-b66a-ba2cfbd01256"><ac:plain-text-body><![CDATA[Update the TLD list in the database. This reads input from IANA ftp site
+from the directory.
+No action is taken on TLDs that are in the database but are not in the input data.
+Usage: database_update_tlds_from_zone options
+Supported options:
+  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
+<h3>Datafiles</h3>
+<h4>datafiles_create_summary</h4>
+<ul>
+<li>This is a useful utility script that can be used to get a high level picture of the processing of the XML files for all the nodes in the system. It can help identify backlogs in the processing system.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="bdc7b96f-0f56-4d9d-a73b-ce13df2d9d2b"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Generate a processing report based on the information in the log files
+
+Usage: datafiles_create_summary options
+
+Supported options:
+	-s Only report on the server with this name (default is all servers)
+	-n Only report on the node with this name (default is all nodes)
+	-c Output in csv format for import into spreadsheet application
+    -d Run in debug mode (set -x)
+	-h Show this help.]]></ac:plain-text-body></ac:structured-macro>
+<h4>datafiles_rm_empty_xml_dirs</h4><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="88bd5715-ba68-4a20-a7f1-be499d770540"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Remove empty incoming xml directories after a certain period of time 
+(default is all the processed directories older than 7 days ago)
+
+Usage: datafiles_rm_empty_xml_dirs options
+
+Supported options:
+	-d Date before which removing empty xml directories 
+      (default: 7 days ago, format YYYY-MM-DD)
+	-h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h4>datafiles_tar_old_xml_files</h4><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="9283cb80-b944-415c-8f1b-065ade9f307f"><ac:plain-text-body><![CDATA[Pack old xml files already processed into a .tar package to store and archive.
+The xml files are packed according to their date and node and server.
+The filename has the following format done-PID-date.tar.bz2 (e.g. done-14143-2015-05-01.tar.bz2).
+
+Usage: datafiles_tar_old_xml_files options
+
+Supported options:
+  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
+<h2>Web front-end</h2>
+<p>These scripts should be run as the DB_READ_USER user (<em>www-data</em> by default).</p>
+<h3>Plotcache</h3>
+<h4>plotcache_generate_cached_plots</h4>
+<ul>
+<li>
+<p>You must add at least one option for which plot window to generate [-D|-W|-M].</p></li>
+<li>
+<p>By default, the cached plots are generated daily for the day before between 00:00 and 23:59. It can be helpful to configure a daily cron job to use this script to create cached plots to make loading of the homepage and common plots faster.&nbsp;</p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="e3a20735-7555-41f6-8f0f-1b885b193085"><ac:rich-text-body>
+<p><span>If not relying on the default behaviour be careful to specify BOTH a start and end date for this script, otherwise the end date will default to today and this may result in many plots being generated.</span></p></ac:rich-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p></li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4a423b5a-ca47-4597-99b9-9e7da0d06d90"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Create cached plots for each chosen period (day/week/month) between Start and End range of time.
+
+Usage: generate_cached_plots options
+
+Supported options:
+    -s Start date of time range from which to create cached plots 
+       (default is 1 day before at 00:00, format YYYY-MM-DDThh:mm)
+    -e End date of time range from which to create cached plots 
+       (default is 1 day before at 23:59, format YYYY-MM-DDThh:mm)
+    -D Generate daily cached plots
+    -W Generate weekly cached plots
+    -M Generate monthly cached plots
+    -h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h4>plotcache_rm_cached_plots</h4>
+<ul>
+<li>
+<p>The user should not normally need to run this script, but is available in case problem are encountered with cached plots, or it is desirable to flush the cache for some reason (for example the default interactive plot type is changed after graphs have been cached).&nbsp;</p></li>
+<li>
+<p>This script is interactive if the -a option is not selected.&nbsp;You will be asked to select which cached plot types you'd like to remove.</p></li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="839813d2-ab5b-4555-a419-880edcf01815"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Remove all or a set of cached plots from the directory structure.
+ 
+Usage: plotcache_rm_cached_plots options
+
+Supported options:
+  -a Remove all cached plots
+  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h3>RSSAC</h3>
+<h4>rssac_generate_reports</h4>
+<ul>
+<li>By default the script creates RSSAC reports for all configured servers for a date 7 days in the past</li>
+<li>It will require that the&nbsp;database_process_rssac_data script will have already been run for the date in question.<em>&nbsp;</em></li>
+<li>Note the unique_source data for the report is taken from the summary table, not the raw table.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="31543074-e163-4040-8ccd-a8ec5c66780c"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Create RSSAC reports for a specified 24 hour period
+ 
+Usage: rssac_generate_reports options
+
+Supported options:
+  -d Date on which to create RSSAC reports 
+     (default is 7 days ago, format: YYYY-MM-DD)
+  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
+<h2>Other tools</h2>
+<h3>conf_read</h3>
+<ul>
+<li>The script parses the contents of the <em>&lt;prefix&gt;/etc/hedgehog/hedgehog.yaml</em> file into environment variables for use my the other scripts. Users will not need to run this directly.</li>
+<li>The database parameters are to be set by the user (if default ones need to be modified) and the directory structure parameters are auto-generated and better left alone.</li></ul>
\ No newline at end of file
diff --git a/docs/src/Hedgehog_Upgrade_Guide_(to_Packges).xml b/docs/src/Hedgehog_Upgrade_Guide_(to_Packges).xml
new file mode 100644
index 0000000..c411673
--- /dev/null
+++ b/docs/src/Hedgehog_Upgrade_Guide_(to_Packges).xml
@@ -0,0 +1,6 @@
+
+<p><br /></p>
+<p>This document describes how to upgrade a server from release 2.4.1 to release 2.4.2 of Hedgehog via packages.&nbsp;</p>
+<h2>Release 2.4.2</h2>
+<p>There are no Hedgehog actions required for this upgrade.</p>
+<p><br /></p>
\ No newline at end of file
diff --git a/docs/src/Hedgehog_User_Guide.xml b/docs/src/Hedgehog_User_Guide.xml
new file mode 100644
index 0000000..0145bb4
--- /dev/null
+++ b/docs/src/Hedgehog_User_Guide.xml
@@ -0,0 +1,630 @@
+
+<p><ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="a93a02e9-93ad-4b13-b26a-5c1cb6b996ce" /></p>
+<h1>Overview of Hedgehog</h1>
+<p>Hedgehog is a visualisation tool for DNS statistics that consumes data&nbsp;acquired with the DSC collector.</p>
+<p>Hedgehog was initially developed for ICANN by Sinodun IT and is now released&nbsp;open source under the MPL License. For more information on the development and support of Hedgehog see the <a href="http://www.dns-stats.org">http://www.dns-stats.org</a>&nbsp;website.</p>
+<h1>Components</h1>
+<p>Hedgehog comprises 3 components which can be run on the same or different machines:</p>
+<ul>
+<li>A Postgres Database</li>
+<li>Data Manager
+<ul>
+<li>Scripts for database creation and management</li>
+<li>XML/DAT processing</li></ul></li>
+<li>Web GUI front-end</li></ul>
+<h2>Installation options</h2>
+<p>Hedgehog can be installed to run on a single server with all the components local to that server.&nbsp;</p>
+<p><ac:image ac:height="400"><ri:attachment ri:filename="Hedgehog_component_view.jpg" /></ac:image></p>
+<p>Alternatively the DB may be on a remote server and the Data Manager and Web front-end may be on different servers as shown below.</p>
+<p><ac:image ac:height="400"><ri:attachment ri:filename="Hedgehog_component_view_split.jpg" /></ac:image></p>
+<h3>Install from Packages (Ubuntu)</h3>
+<p>From 2.2 packages for Ubuntu are provided for Hedgehog. It is recommended to use this method for new installs.</p>
+<h3>Install from Source Code</h3>
+<p>Hedgehog can also be installed directly from source code if desired - note: this is not a supported method.</p>
+<h1>Datasets</h1><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="1dccef1c-27f8-4b28-84b4-ca7010c69dc6"><ac:rich-text-body>
+<p>Hedgehog is currently tested against the output of the original DSC collector. It has not been tested against the recently released DSC 2.0.</p></ac:rich-text-body></ac:structured-macro>
+<p>&nbsp;</p>
+<p>An example dsc.conf file for the data sets that Hedgehog supports is given below:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4b93a047-e94d-4b6b-a945-fb6e5ec2414a"><ac:plain-text-body><![CDATA[# Basic data sets
+dataset qtype dns All:null Qtype:qtype queries-only;
+dataset rcode dns All:null Rcode:rcode replies-only;
+dataset opcode dns All:null Opcode:opcode queries-only;
+dataset rcode_vs_replylen dns Rcode:rcode ReplyLen:msglen replies-only;
+dataset client_subnet dns All:null ClientSubnet:cip4_net queries-only max-cells=200;
+dataset qtype_vs_qnamelen dns Qtype:qtype QnameLen:qnamelen queries-only;
+dataset qtype_vs_tld dns Qtype:qtype TLD:tld queries-only,popular-qtypes max-cells=200;
+dataset certain_qnames_vs_qtype dns CertainQnames:certain_qnames Qtype:qtype queries-only;
+dataset client_subnet2 dns Class:query_classification ClientSubnet:cip4_net queries-only max-cells=200;
+dataset client_addr_vs_rcode dns Rcode:rcode ClientAddr:client replies-only max-cells=50;
+dataset chaos_types_and_names dns Qtype:qtype Qname:qname chaos-class,queries-only;
+dataset idn_qname dns All:null IDNQname:idn_qname queries-only;
+dataset edns_version dns All:null EDNSVersion:edns_version queries-only;
+dataset do_bit dns All:null D0:do_bit queries-only;
+dataset rd_bit dns All:null RD:rd_bit queries-only;
+dataset ipv6_rsn_abusers dns All:null ClientAddr:client queries-only,aaaa-or-a6-only,root-servers-net-only max-cells=50;
+dataset transport_vs_qtype dns Transport:transport Qtype:qtype queries-only;
+dataset direction_vs_ipproto ip Direction:ip_direction IPProto:ip_proto any;
+dataset dns_ip_version_vs_qtype dns IPVersion:dns_ip_version Qtype:qtype queries-only;]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="eb6a3061-2017-49ee-aad8-6b96ad729e08"><ac:plain-text-body><![CDATA[# Additional data sets for RSSAC
+dataset unique_sources dns IPVersion:dns_ip_version ClientAddr:client queries-only; 
+dataset traffic_volume_queries dns Transport:transport IPVersion:dns_ip_version queries-only; 
+dataset traffic_volume_responses dns Transport:transport IPVersion:dns_ip_version replies-only; 
+dataset traffic_sizes_queries dns Transport:transport MsgLen:msglen queries-only; 
+dataset traffic_sizes_responses dns Transport:transport MsgLen:msglen replies-only; ]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4eaba3e3-a548-4198-89ac-4dfc069be91f"><ac:plain-text-body><![CDATA[# New in 2.1
+dataset server_addr dns All:null ServerAddr:server queries-only;
+# New in 2.2.0
+dataset qtype_vs_client_subnet dns Qtype:qtype ClientSubnet:cip4_net queries-only max-cells=200;]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="7fd11eb1-b417-4b2a-8095-555c12a64e33"><ac:plain-text-body><![CDATA[# New in 2.4.0
+dataset server_addr_vs_trans dns Transport:transport ServerAddr:server queries-only;]]></ac:plain-text-body></ac:structured-macro>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="26a887fa-d2f8-4988-9415-ab76c5e22739"><ac:rich-text-body>
+<p>Data collection for the server_addr dataset is not yet supported in the official release of version 1 of DSC, but is available using the latest code from the DSC github repo. Also, note this dataset will contain the source addresses of all responses (QR = 1) seen on the collection node even if they do not originate from the local_address specified in the dsc.conf (i.e. if another server sends a response to the collection node for some reason).</p></ac:rich-text-body></ac:structured-macro>
+<p>&nbsp;</p>
+<h1>Plots</h1>
+<p>The relationship between the Hedgehog plots and those produced by the DSC Presenter are listed here for convenience.</p>
+<table><colgroup><col /><col /><col /><col /></colgroup>
+<tbody>
+<tr>
+<th>Hedgehog plot category</th>
+<th colspan="1">Hedgehog plot display name</th>
+<th>DSC (and Hedgehog internal) plot id</th>
+<th>DSC display name</th></tr>
+<tr>
+<td colspan="1"><strong>Node Statistics</strong></td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">By node</td>
+<td colspan="1">by_node</td>
+<td colspan="1">By node</td></tr>
+<tr>
+<td>
+<p class="p2"><span class="s1"><strong>Query Attributes</strong></span></p></td>
+<td colspan="1">
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">
+<p class="p2"><span class="s1">CHAOS queries</span></p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">chaos_types_and_names</span></p></td>
+<td colspan="1">CHAOS</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">DO bit</span></p></td>
+<td>
+<p class="p2"><span class="s1">do_bit</span></p></td>
+<td>Query Attributes &gt; DO bit</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">EDNS version</span></p></td>
+<td>
+<p class="p2"><span class="s1">edns_version</span></p></td>
+<td>Query Attributes &gt; EDNS version</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">IDN qnames</span></p></td>
+<td>
+<p class="p2"><span class="s1">idn_qname</span></p></td>
+<td>Query Attributes &gt; IDN Qnames</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">OPCODE</span></p></td>
+<td>
+<p class="p2"><span class="s1">opcode</span></p></td>
+<td>Opcodes</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">
+<p class="p2"><span class="s1">RD bit</span></p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">rd_bit</span></p></td>
+<td colspan="1">Query Attributes &gt; RD bit</td></tr>
+<tr>
+<td>
+<p class="p2"><span class="s1"><strong>QTYPE</strong></span></p></td>
+<td colspan="1">&nbsp;</td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>&nbsp;</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">DNS queries by QTYPE</span></p></td>
+<td>
+<p class="p2"><span class="s1">qtype</span></p></td>
+<td>Qtypes</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">DNSSEC queries by QTYPE</span></p></td>
+<td>
+<p class="p2"><span class="s1">dnssec_qtype</span></p></td>
+<td>DNSSEC Qtypes</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">
+<p class="p2"><span class="s1">Popular query names by QTYPE</span></p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">certain_qnames_vs_qtype</span></p></td>
+<td colspan="1">Popular Names</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">
+<p class="p2"><span class="s1">QTYPE for most popular TLDs</span></p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">qtype_vs_tld</span></p></td>
+<td colspan="1">TLDs</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">Query Name Lengths by QTYPE</span></p></td>
+<td>
+<p class="p2"><span class="s1">qtype_vs_qnamlen</span></p></td>
+<td>Qname Lengths</td></tr>
+<tr>
+<td>
+<p class="p2"><span class="s1"><strong>RCODE</strong></span></p></td>
+<td colspan="1">
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>&nbsp;</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">Replies by RCODE</span></p></td>
+<td>
+<p class="p2"><span class="s1">rcode</span></p></td>
+<td>Rcodes</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">Reply lengths by RCODE </span></p></td>
+<td>
+<p class="p2"><span class="s1">rcode_vs_replylen</span></p></td>
+<td>Reply Lengths</td></tr>
+<tr>
+<td>
+<p class="p2"><span class="s1"><strong>IP Protocol</strong></span></p></td>
+<td colspan="1">
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>&nbsp;</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">IP version</span></p></td>
+<td>
+<p class="p2"><span class="s1">dns_ip_version</span></p></td>
+<td>IP version</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">
+<p class="p2"><span class="s1">Queries by IP version, QTYPE</span></p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">dns_ip_version_vs_qtype</span></p></td>
+<td colspan="1">IP version &gt; Query types</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">Received packets by IP protocol</span></p></td>
+<td>
+<p class="p2"><span class="s1">direction_vs_ipproto</span></p></td>
+<td>IP protocols</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p2"><span class="s1">Transports carrying DNS queries</span></p></td>
+<td>
+<p class="p2"><span class="s1">transport_vs_qtype</span></p></td>
+<td>DNS transport</td></tr>
+<tr>
+<td colspan="1"><strong>PCAP statistics</strong></td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">PCAP statistics</td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td>
+<p class="p2"><span class="s1"><strong>Client Subnet Statistics</strong></span></p></td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>&nbsp;</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p2"><span class="s1">Busiest client subnets</span></p></td>
+<td>
+<p class="p2"><span class="s1">client_subnet_accum</span></p></td>
+<td>Client Geography</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p2"><span class="s1">RCODE by client subnet</span></p></td>
+<td>
+<p class="p2"><span class="s1">client_addr_vs_rcode_accum</span></p></td>
+<td>Rcodes by Client Address</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p2"><span class="s1">Root abusers</span></p></td>
+<td>
+<p class="p2"><span class="s1">ipv6_rsn_abusers_accum</span></p></td>
+<td>IPv6 root abusers</td></tr>
+<tr>
+<td>
+<p class="p2"><span class="s1"><strong>Classification</strong></span></p></td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>&nbsp;</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p2"><span class="s1">Query classification by client subnet (accum)</span></p></td>
+<td>
+<p class="p2"><span class="s1">client_subnet2_accum</span></p></td>
+<td>Classification</td></tr>
+<tr>
+<td>
+<p class="p1">&nbsp;</p></td>
+<td>
+<p class="p2"><span class="s1">Query classification by client subnet (count)</span></p></td>
+<td>
+<p class="p2"><span class="s1">client_subnet2_count</span></p></td>
+<td>Classification &gt; count</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">
+<p class="p2"><span class="s1">Query classifications</span></p></td>
+<td colspan="1">
+<p class="p2"><span class="s1">client_subnet2_trace</span></p></td>
+<td colspan="1">Classification &gt; trace</td></tr></tbody></table>
+<p>In addition Hedgehog is capable of displaying plots based on datasets defined in the RSSAC 002 document (see later for more details):</p>
+<table>
+<tbody>
+<tr>
+<th>Hedgehog plot category</th>
+<th>Hedgehog plot display name</th>
+<th colspan="1"><span>Hedgehog internal plot id</span></th></tr>
+<tr>
+<td><strong>RSSAC</strong></td>
+<td>&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1"><span>RCODE volume</span></td>
+<td colspan="1">rcode_volume</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Traffic sizes</td>
+<td colspan="1">traffic_sizes</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Traffic volume</td>
+<td colspan="1">traffic_volume</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Traffic volume differences</td>
+<td colspan="1">traffic_volume_differences</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Unique sources</td>
+<td colspan="1">unique_sources</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Zone size</td>
+<td colspan="1">zones_size</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Zone propagation time</td>
+<td colspan="1">load_time</td></tr></tbody></table>
+<p>The following plots are new in 2.1</p>
+<table>
+<tbody>
+<tr>
+<th>Hedgehog plot category</th>
+<th>Hedgehog plot display name</th>
+<th colspan="1"><span>Hedgehog internal plot id</span></th></tr>
+<tr>
+<td><strong>Node Statistics</strong></td>
+<td>&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td>&nbsp;</td>
+<td>
+<p>By instance</p></td>
+<td colspan="1">by_instance</td></tr>
+<tr>
+<td>&nbsp;</td>
+<td>By server IP address</td>
+<td colspan="1">server_addr</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">By city</td>
+<td colspan="1">by_city</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">By country</td>
+<td colspan="1">by_country</td></tr>
+<tr>
+<td colspan="1"><strong>QTYPE vs TLS</strong></td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">QTYPE for most popular Undelegated TLDs</td>
+<td colspan="1">qtype_vs_othertld</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">QTYPE for most popular ccTLDs</td>
+<td colspan="1"><span>qtype_vs_cctld</span></td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">QTYPE for most popular Legacy TLDs</td>
+<td colspan="1"><span>qtype_vs_legacytld</span></td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">QTYPE for most popular New TLDs</td>
+<td colspan="1"><span>qtype_vs_newtld</span></td></tr>
+<tr>
+<td colspan="1"><strong>Client Subnet Statistics</strong></td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Busiest client ASNs &nbsp;</td>
+<td colspan="1">asn</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Busiest client BGP prefix</td>
+<td colspan="1">bgpprefix</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">RCODE by client ASNs &nbsp;</td>
+<td colspan="1">
+<p class="p1"><span class="s1">client_addr_vs_rcode_accum_asn</span></p></td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">RCODE by clients by BGP prefix&nbsp;</td>
+<td colspan="1">
+<p class="p1"><span class="s1">client_addr_vs_rcode_accum_bgpprefix</span></p></td></tr>
+<tr>
+<td colspan="1"><strong>Classification</strong></td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Query classification by clients by BGP prefix</td>
+<td colspan="1">
+<p class="p1"><span class="s1">client_subnet2_accum_bgpprefix&nbsp;</span></p></td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Query classification by client ASNs</td>
+<td colspan="1">
+<p class="p1"><span class="s1">client_subnet2_accum_asn</span></p></td></tr>
+<tr>
+<td colspan="1"><strong>RSSAC</strong></td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">
+<p>Zone size</p></td>
+<td colspan="1">zone_size</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Zone propagation time</td>
+<td colspan="1">load_time</td></tr>
+<tr>
+<td colspan="1"><strong>GEO</strong></td>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Busiest client locations by country&nbsp;</td>
+<td colspan="1">geochart</td></tr>
+<tr>
+<td colspan="1">&nbsp;</td>
+<td colspan="1">Busiest client locations by city</td>
+<td colspan="1">geomap</td></tr></tbody></table>
+<p>The following plots are new in 2.2</p>
+<table>
+<tbody>
+<tr>
+<th>Hedgehog plot category</th>
+<th>Hedgehog plot display name</th>
+<th colspan="1"><span>Hedgehog internal plot id</span></th></tr>
+<tr>
+<td><strong>Client Subnet Statistics</strong></td>
+<td>&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td>&nbsp;</td>
+<td>
+<p>DNSKEY Queries by ASN</p></td>
+<td colspan="1">
+<p class="p1"><span class="s1">dnskey_vs_asn</span></p></td></tr></tbody></table>
+<p>The following plots are new in 2.3</p>
+<table>
+<tbody>
+<tr>
+<th>Hedgehog plot category</th>
+<th>Hedgehog plot display name</th>
+<th colspan="1">Hedgehog internal plot id</th></tr>
+<tr>
+<td><strong>Node Statistics</strong></td>
+<td>&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td>&nbsp;</td>
+<td>
+<p>By region</p></td>
+<td colspan="1">
+<p class="p1"><span class="s1">by_region</span></p>
+<div><span class="s1"><br /></span></div></td></tr></tbody></table>
+<p>The following plots are new in 2.4</p>
+<table>
+<tbody>
+<tr>
+<th>Hedgehog plot category</th>
+<th>Hedgehog plot display name</th>
+<th colspan="1">Hedgehog internal plot id</th></tr>
+<tr>
+<td><strong>Server IP</strong></td>
+<td>&nbsp;</td>
+<td colspan="1">&nbsp;</td></tr>
+<tr>
+<td>&nbsp;</td>
+<td>
+<p>By server IP address and transport</p></td>
+<td colspan="1">
+<p class="p1">server_addr_vs_trans</p></td></tr></tbody></table>
+<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="f7a8bc07-2c7e-482e-850b-ad49fc4eae75"><ac:rich-text-body>
+<ul>
+<li><strong>RSSAC: Zone size and Zone propagation time graphs.</strong> In order for data to be available for these graphs, an instance of the <em>rssacd</em><span>&nbsp;demon&nbsp;must be run to collect the data (it is not provided by the DSC collector).</span></li>
+<li><strong>GEO: Busiest client location by country.</strong> Both client locations and collection node locations can be displayed on this graph. For node locations to be displayed the nodes must have a city specified via the nodes.csv file.</li></ul></ac:rich-text-body></ac:structured-macro>
+<h1>Directory Structure</h1>
+<p>If installed from source code Hedgehog uses the following directory structure under the install &lt;prefix&gt; directory (/usr/local/ by default)</p>
+<p><ac:image ac:width="550"><ri:attachment ri:filename="Hedgehog_directory_structure.jpg" /></ac:image></p>
+<p>When installing from a Ubuntu package the structure is different:</p>
+<table>
+<thead>
+<tr>
+<th>Location (install from source)</th>
+<th>
+<div class="tablesorter-header-inner">Location (install from package)</div></th>
+<th colspan="1">
+<div class="tablesorter-header-inner">Contents</div></th></tr></thead>
+<tbody>
+<tr>
+<td>&lt;prefix&gt;/etc/hedgehog/</td>
+<td>/etc/hedgehog/</td>
+<td colspan="1">Hedgehog configuration files (yaml and nodes)</td></tr>
+<tr>
+<td colspan="1">&lt;prefix&gt;/etc/hedgehog/*.sample</td>
+<td colspan="1">/usr/share/doc/hedgehog/examples/*.sample</td>
+<td colspan="1">Sample Hedgehog configuration files</td></tr>
+<tr>
+<td>&lt;prefix&gt;/var/hedgehog/</td>
+<td>/var/lib/hedgehog/</td>
+<td colspan="1">
+<p>'data' directory containing XML data</p>
+<p>'www' directory containing cached plots and rssac reports</p></td></tr>
+<tr>
+<td colspan="1">&lt;prefix&gt;/bin/hedgehogctl</td>
+<td colspan="1">/usr/bin/hedgehogctl</td>
+<td colspan="1">Wrapper script for Hedgehog tools</td></tr>
+<tr>
+<td colspan="1">&lt;prefix&gt;/bin/refile_and_grok</td>
+<td colspan="1">/usr/bin/refile_and_grok</td>
+<td colspan="1">refile_and_grok script</td></tr>
+<tr>
+<td colspan="1">&lt;prefix&gt;/libexec/hedgehog/</td>
+<td colspan="1">/usr/lib/hedgehog/</td>
+<td colspan="1">Hedgehog tools scripts and DDL updates</td></tr></tbody></table>
+<p>In the rest of this document only the relative path under the install location is given so the instructions are applicable to both methods of install.</p>
+<h1>Tools</h1>
+<p>Hedgehog provides a set of Data Manager scripts to support the database and data file management, and Web scripts to generate cached plots and the RSSAC reports. They are documented separately in the Hedgehog Tools PDF.&nbsp;</p>
+<h1>Data Manager</h1>
+<h2>Upload of Data</h2>
+<p>It is assumed that the user has already deployed an upload mechanism used with the DSC collector and presenter. A common practice is to use ssh for the upload.</p>
+<p>Hedgehog also supports the use of WebDAV for uploads, validated with certificates. This is particularly useful for deployments with many nodes as it scales better than ssh. It can be configured with the hedgehog-webdav-upload package.</p>
+<h2><span style="line-height: 1.25;">Import of data</span></h2>
+<p>Hedgehog provides a refile_and_grok script for this purpose. See the Installation guide for more details of how to configure this.</p>
+<h2>Monitoring data import</h2>
+<p>The import process produces two different logs for the user to track the progress of the data import:</p>
+<ul>
+<li>The<em> refile_and_grok</em>&nbsp;script writes output about which nodes it is processing to a <em>refile_and_grok.out</em> file in the Hedgehog <em>/var/hedgehog/data</em> directory. This gives a high level view of the processing.&nbsp;<br />&nbsp;</li>
+<li>The import also generates a log file per node in the <em>/var/hedgehog/data/&lt;server&gt;/&lt;node&gt;</em>&nbsp;directory called <em>dsc-extractor.out</em>. This lists more details on the processing of each XML file for a specific node.&nbsp;<br />&nbsp;</li>
+<li>A utility script is also provided: <em>/bin/hedgehogctl datafiles_create_summary</em>&nbsp;which can be run to get an summary of the current state of processing.</li></ul>
+<h2>Adding or updating servers and nodes</h2>
+<p>If new servers or nodes need to be added to the system then the user should:</p>
+<ul>
+<li>update the <em>/etc/hedgehog/nodes.csv</em> files as required to include the new servers/nodes<br />&nbsp;</li>
+<li>run the&nbsp;<em>/bin/hedgehogctl database_update_nodes</em>&nbsp;script which will update the data in the database and create any required directory structures</li></ul><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="049c0d31-3cdd-4ef8-ac3c-ef5ad4e0e487"><ac:rich-text-body>
+<p>In the event of adding new servers the user&nbsp;<strong>must</strong>&nbsp;also then run the&nbsp;<em>/bin/hedgehogctl database_manage_partition</em> scripts to create the required database tables so that data can be imported for the new server.&nbsp;</p></ac:rich-text-body></ac:structured-macro>
+<h1>Web front-end</h1>
+<h2>Plot caching</h2>
+<p>Hedgehog caches plot image files as they are generated by the user. The plot files are cached in the directory listed in the <em>/etc/hedgehog/hedgehog.yaml</em> file in the directories section for 'plotcache' (this path is auto-generated in 2.0 and should not be changed by the user).</p>
+<ul>
+<li>The reuse of cached plots can be enabled/disabled by changing the <em>use_plot_caching</em>&nbsp;option in the /<em>share/www/conf/hedgehog_gui.yaml</em> configuration file<br />&nbsp;</li>
+<li>The <em>caching_delay_in_hours</em>&nbsp;option in the same file controls the delay applied to caching the most recently uploaded data. For example the default for this is 1 i.e. only plots for which the end time is more than one hour ago will be cached. This avoids caching plots where the data may still be being imported for some nodes if there were a delay in upload or processing for some reason.&nbsp;<br />&nbsp;</li>
+<li>If desired, a cron job can be created that will create cached plots for specific time periods, for example it may be useful to have a cron job to create daily plots for the previous day (see the section on Maintenance). This will make loading of the homepage faster.&nbsp;<br />&nbsp;</li>
+<li>If needed the cached files for selected or all plots may be deleted by running the<em>&nbsp;/bin/hedgehogctl plotcache_rm_cached_plots</em>&nbsp;script.&nbsp;</li></ul>
+<h2>RSSAC reports</h2>
+<h3>Data collection</h3>
+<p>If the required datasets are collected via the DSC collector (see Datasets above), Hedgehog can produce RRSAC reports as described in the &quot;<span style="line-height: 1.4285715;">&nbsp;</span><a href="https://www.icann.org/en/system/files/files/rssac-002-measurements-root-20nov14-en.pdf"><span style="line-height: 1.4285715;">RSSAC Recommendation on Measurements of the Root Server System -&nbsp;</span><span style="line-height: 1.4285715;">RSSAC 002</span></a><span style="line-height: 1.4285715;">&nbsp;</span>&quot; document for the following metrics:</p>
+<ul>
+<li>traffic-volume</li>
+<li>traffic-size</li>
+<li>rcode-volume</li>
+<li>unique-sources</li>
+<li>zone-size</li>
+<li>load-time</li></ul>
+<h3>RSSACD</h3>
+<p>In order to collect the zone-size and load-time statistics a dedicated <em>rssacd</em> demon must be run on the Hedgehog Data Manager to collect the information directly from the nodes. It requires that the IP addresses of the nodes have been populated via the nodes.csv file. More details are given in the Installation/Upgrade Guides.</p>
+<h3>Generating the reports</h3>
+<ul>
+<li>To produce the RSSAC reports 2 scripts must be run <span>(see the&nbsp;Hedgehog Tools guide for more information)</span>:
+<ol>
+<li><span>the&nbsp;</span><em>&lt;prefix&gt;/bin/hedgehogctl database_process_rssac_data</em><span>&nbsp;script via the Database tools component</span></li>
+<li>the <em>&lt;prefix&gt;/bin/hedgehogctl rssac_generate_reports</em>&nbsp;script on the Web front-end component that will store and display the results&nbsp;</li></ol></li>
+<li><span>The resulting yaml files (along with matching plot files) are stored in the directory listed in the&nbsp;</span><em>&lt;prefix&gt;/etc/hedgehog/hedgehog.yaml</em><span>&nbsp;file in the directories section for '</span><em>rssac</em><span>' (this path is auto-generated in 2.0 and should not be changed by the user).&nbsp;</span></li>
+<li>The generated files are served by the Hedgehog front end and can be found by clicking on the &quot;RSSAC reports&quot; link at the bottom left of the Hedgehog webpage.&nbsp;</li></ul><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="515cb692-5b1f-4ad1-8da2-56862ef2a46f"><ac:rich-text-body>
+<p>Note that RSSAC reports can be run for any server, however the name of the server affects the value in the service field of the generated yaml: &nbsp;</p>
+<p>If the server name is of the pattern &quot;*-root&quot; where * is a letter, then the service will be &quot;*.<a href="http://root-servers.org">root-servers.org</a>&quot;. For all other server names the service will simply be the server name.&nbsp;</p></ac:rich-text-body></ac:structured-macro>
+<p>The script&nbsp;can either be run manually, or via a cron job as described in the Maintenance section of the installation guide.</p>
+<h3>Notes</h3>
+<p>Some notes on the RSSAC data:</p>
+<ul>
+<li>The DSC collector cannot collect extended RCODES and therefore Hedgehog cannot include this information in the rcode_volume report<br />&nbsp;</li>
+<li>As a result of the DSC collector mechanism (i.e collecting data in 1 minute windows) the total quantity of data collected in the 'unique_sources' dataset can be significant and can result in exceptionally large database tables for this dataset. However, this 'raw' data must be uploaded to the Hedgehog database so that it can be accumulated over the 24 hour report period.<br /><br />
+<ul>
+<li>Once the raw data for the entire 24 hours is uploaded, the RSSAC report can be generated to create the summary figures required in the report and the 'raw' data may be discarded to save disk space (see the Maintanance section of the Installation guide for details of the script to do this).<br />&nbsp;</li>
+<li>The 'Unique sources' plot in the Hedgehog GUI is generated from the summary data, not from the raw data and therefore the plot cannot be viewed until after the relevant RSSAC report has been generated via the <em>rssac_generate_reports</em> script.&nbsp;</li></ul></li></ul>
+<h1>Limitations/Known issues</h1>
+<p>Users should be aware of several limitations in the current version of Hedgehog as listed below. It is hoped they will be addressed in future versions</p>
+<ul>
+<li>&nbsp;The full list of possible plots are displayed via the drop down menu in the GUI, regardless of whether there is data in the database for the plot.<br />&nbsp;</li>
+<li>
+<p>There is a known issue with the SVG line plots where long legends are not wrapped properly:<br /><a href="https://code.google.com/p/google-visualization-api-issues/issues/detail?id=1536">https://code.google.com/p/google-visualization-api-issues/issues/detail?id=1536</a>&nbsp;. This affects, for example, the 'by node' plot if there are many nodes.&nbsp;<span>&nbsp;A configuration option is available (</span><em>default_interactive_plot_type</em><span>&nbsp;in&nbsp;</span><em>hedgehog_gui</em><span>.yaml) which can specify the system to produce non-SVG line plots that do not have this issue - however these plots require Flash.<br /><br /></span></p></li>
+<li>No static plots are provided for the GEO plot category. Plotting of the GEO &quot;Busiest client locations by country' plot is dynamic and may be slow if there are many countries to be plotted.&nbsp;<br />&nbsp;</li>
+<li>Server, node and group and instance names may only contain alphanumeric characters, full stops (.) and hyphens (-).<span>&nbsp;Country and city names can only contain&nbsp;<span>alphanumeric characters, spaces and hyphens (-)</span></span><br />&nbsp;</li>
+<li>The Unique Sources plot is not available until after the RSSAC report has been generated for the date in question.&nbsp;</li></ul>
+<p>&nbsp;</p>
+<p>&nbsp;</p>
+<p>&nbsp;</p>
\ No newline at end of file
diff --git a/docs/src/Hedgehog_component_view.png b/docs/src/Hedgehog_component_view.png
new file mode 100644
index 0000000..fccf550
Binary files /dev/null and b/docs/src/Hedgehog_component_view.png differ
diff --git a/docs/src/Hedgehog_component_view_split.png b/docs/src/Hedgehog_component_view_split.png
new file mode 100644
index 0000000..13f93ab
Binary files /dev/null and b/docs/src/Hedgehog_component_view_split.png differ
diff --git a/docs/src/Hedgehog_directory_structure.png b/docs/src/Hedgehog_directory_structure.png
new file mode 100644
index 0000000..6fe84a4
Binary files /dev/null and b/docs/src/Hedgehog_directory_structure.png differ
diff --git a/m4/acx_components.m4 b/m4/acx_components.m4
new file mode 100644
index 0000000..76f2070
--- /dev/null
+++ b/m4/acx_components.m4
@@ -0,0 +1,22 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+AC_DEFUN([ACX_COMPONENTS],
+[
+    AC_ARG_ENABLE([web],
+                AS_HELP_STRING([--disable-web],
+                [Don't build the web interface - enabled by default.]),
+                [AS_IF([test "x$enableval" == "xno"], [WEB=0])])
+                
+    AC_ARG_ENABLE([data-manager],
+                AS_HELP_STRING([--disable-data-manager],
+                [Don't build the data manager - enabled by default.]),
+                [AS_IF([test "x$enableval" == "xno"], [DM=0])])
+
+])
\ No newline at end of file
diff --git a/m4/acx_libpostgres.m4 b/m4/acx_libpostgres.m4
new file mode 100644
index 0000000..17f4449
--- /dev/null
+++ b/m4/acx_libpostgres.m4
@@ -0,0 +1,48 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+AC_DEFUN([ACX_POSTGRES],[
+	AC_ARG_WITH([postgres], 
+		[AC_HELP_STRING([--with-postgres=PATH],[specify prefix of path of PostgreSQL])],
+        	[
+			POSTGRES_PATH="$withval"
+		],[
+			POSTGRES_PATH="/usr/local"
+		])
+
+        AC_ARG_WITH([postgres-includes], 
+                [AC_HELP_STRING([--with-postgres-includes=PATH],[specify include prefix of PostgreSQL])],
+                [
+                        POSTGRES_INCLUDE_PATH="$withval"
+                ],[
+                        POSTGRES_INCLUDE_PATH="/usr/include"
+                ])
+
+
+	AC_MSG_CHECKING(what are the postgres includes)
+	POSTGRES_INCLUDES="-I$POSTGRES_INCLUDE_PATH"
+	AC_MSG_RESULT($POSTGRES_INCLUDES)
+
+	AC_MSG_CHECKING(what are the postgres libs)
+	POSTGRES_LIBS="-L$POSTGRES_PATH/lib64 -L$POSTGRES_PATH/lib -lpq"
+	AC_MSG_RESULT($POSTGRES_LIBS)
+
+	tmp_CFLAGS=$CFLAGS
+	tmp_LIBS=$LIBS
+
+	CFLAGS="$CFLAGS $POSTGRES_INCLUDES"
+	LIBS="$LIBS $POSTGRES_LIBS"
+
+	AC_CHECK_LIB(pq, PQconnectdb,,[AC_MSG_ERROR([Can't find PostgreSQL])])
+	LIBS=$tmp_LIBS
+	CFLAGS=$tmp_CFLAGS
+
+	AC_SUBST(POSTGRES_INCLUDES)
+	AC_SUBST(POSTGRES_LIBS)
+])
diff --git a/m4/acx_libpqxx.m4 b/m4/acx_libpqxx.m4
new file mode 100644
index 0000000..b8e176d
--- /dev/null
+++ b/m4/acx_libpqxx.m4
@@ -0,0 +1,48 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+AC_DEFUN([ACX_LIBPQXX],[
+	AC_ARG_WITH([libpqxx], 
+		[AC_HELP_STRING([--with-libpqxx=PATH],[specify prefix of path of libpqxx])],
+      	[
+			LIBPQXX_PATH="$withval"
+		],[
+			LIBPQXX_PATH="/usr/local"
+		])
+
+	AC_MSG_CHECKING(what are the libpqxx includes)
+	LIBPQXX_INCLUDES="-I$LIBPQXX_PATH/include"
+	AC_MSG_RESULT($LIBPQXX_INCLUDES)
+
+	AC_MSG_CHECKING(what are the libpqxx libs)
+	LIBPQXX_LIBS="-L$LIBPQXX_PATH/lib64 -L$LIBPQXX_PATH/lib -lpqxx"
+	AC_MSG_RESULT($LIBPQXX_LIBS)
+
+        tmp_CFLAGS=$CFLAGS
+        tmp_CXXFLAGS=$CXXFLAGS
+	tmp_LIBS=$LIBS
+
+	CFLAGS="$CFLAGS $LIBPQXX_INCLUDES"
+	CXXFLAGS="$CXXFLAGS $LIBPQXX_INCLUDES"
+	LIBS="$LIBS $LIBPQXX_LIBS"
+
+	AC_LANG(C++)
+        AC_LINK_IFELSE(
+                [AC_LANG_PROGRAM([#include "pqxx/pqxx"],
+                  [pqxx::connection db_conn; pqxx::work db_trans( db_conn ); pqxx::pipeline db_pipe(db_trans)])],
+                  ,
+                  [AC_MSG_ERROR([Can't find libpqxx])])
+
+	LIBS=$tmp_LIBS
+	CFLAGS=$tmp_CFLAGS
+        CXXFLAGS=$tmp_CXXFLAGS
+
+	AC_SUBST(LIBPQXX_INCLUDES)
+	AC_SUBST(LIBPQXX_LIBS)
+])
diff --git a/m4/boost.m4 b/m4/boost.m4
new file mode 100644
index 0000000..8e561a9
--- /dev/null
+++ b/m4/boost.m4
@@ -0,0 +1,1417 @@
+# boost.m4: Locate Boost headers and libraries for autoconf-based projects.
+# Copyright (C) 2007-2011, 2014  Benoit Sigoure <tsuna@lrde.epita.fr>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Additional permission under section 7 of the GNU General Public
+# License, version 3 ("GPLv3"):
+#
+# If you convey this file as part of a work that contains a
+# configuration script generated by Autoconf, you may do so under
+# terms of your choice.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+m4_define([_BOOST_SERIAL], [m4_translit([
+# serial 24
+], [#
+], [])])
+
+# Original sources can be found at http://github.com/tsuna/boost.m4
+# You can fetch the latest version of the script by doing:
+#   wget http://github.com/tsuna/boost.m4/raw/master/build-aux/boost.m4
+
+# ------ #
+# README #
+# ------ #
+
+# This file provides several macros to use the various Boost libraries.
+# The first macro is BOOST_REQUIRE.  It will simply check if it's possible to
+# find the Boost headers of a given (optional) minimum version and it will
+# define BOOST_CPPFLAGS accordingly.  It will add an option --with-boost to
+# your configure so that users can specify non standard locations.
+# If the user's environment contains BOOST_ROOT and --with-boost was not
+# specified, --with-boost=$BOOST_ROOT is implicitly used.
+# For more README and documentation, go to http://github.com/tsuna/boost.m4
+# Note: THESE MACROS ASSUME THAT YOU USE LIBTOOL.  If you don't, don't worry,
+# simply read the README, it will show you what to do step by step.
+
+m4_pattern_forbid([^_?(BOOST|Boost)_])
+
+
+# _BOOST_SED_CPP(SED-PROGRAM, PROGRAM,
+#                [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND])
+# --------------------------------------------------------
+# Same as AC_EGREP_CPP, but leave the result in conftest.i.
+#
+# SED-PROGRAM is *not* overquoted, as in AC_EGREP_CPP.  It is expanded
+# in double-quotes, so escape your double quotes.
+#
+# It could be useful to turn this into a macro which extracts the
+# value of any macro.
+m4_define([_BOOST_SED_CPP],
+[AC_LANG_PUSH([C++])dnl
+AC_LANG_PREPROC_REQUIRE()dnl
+AC_REQUIRE([AC_PROG_SED])dnl
+AC_LANG_CONFTEST([AC_LANG_SOURCE([[$2]])])
+AS_IF([dnl eval is necessary to expand ac_cpp.
+dnl Ultrix and Pyramid sh refuse to redirect output of eval, so use subshell.
+dnl Beware of Windows end-of-lines, for instance if we are running
+dnl some Windows programs under Wine.  In that case, boost/version.hpp
+dnl is certainly using "\r\n", but the regular Unix shell will only
+dnl strip `\n' with backquotes, not the `\r'.  This results in
+dnl boost_cv_lib_version='1_37\r' for instance, which breaks
+dnl everything else.
+dnl Cannot use 'dnl' after [$4] because a trailing dnl may break AC_CACHE_CHECK
+dnl
+dnl Beware that GCC 5, when expanding macros, may embed # line directives
+dnl a within single line:
+dnl
+dnl # 1 "conftest.cc"
+dnl # 1 "<built-in>"
+dnl # 1 "<command-line>"
+dnl # 1 "conftest.cc"
+dnl # 1 "/opt/local/include/boost/version.hpp" 1 3
+dnl # 2 "conftest.cc" 2
+dnl boost-lib-version =
+dnl # 2 "conftest.cc" 3
+dnl                    "1_56"
+dnl
+dnl So get rid of the # lines, and glue the remaining ones together.
+(eval "$ac_cpp conftest.$ac_ext") 2>&AS_MESSAGE_LOG_FD |
+  grep -v '#' |
+  tr -d '\r' |
+  tr -s '\n' ' ' |
+  $SED -n -e "$1" >conftest.i 2>&1],
+  [$3],
+  [$4])
+rm -rf conftest*
+AC_LANG_POP([C++])dnl
+])# _BOOST_SED_CPP
+
+
+
+# BOOST_REQUIRE([VERSION], [ACTION-IF-NOT-FOUND])
+# -----------------------------------------------
+# Look for Boost.  If version is given, it must either be a literal of the form
+# "X.Y.Z" where X, Y and Z are integers (the ".Z" part being optional) or a
+# variable "$var".
+# Defines the value BOOST_CPPFLAGS.  This macro only checks for headers with
+# the required version, it does not check for any of the Boost libraries.
+# On # success, defines HAVE_BOOST.  On failure, calls the optional
+# ACTION-IF-NOT-FOUND action if one was supplied.
+# Otherwise aborts with an error message.
+AC_DEFUN([BOOST_REQUIRE],
+[AC_REQUIRE([AC_PROG_CXX])dnl
+AC_REQUIRE([AC_PROG_GREP])dnl
+echo "$as_me: this is boost.m4[]_BOOST_SERIAL" >&AS_MESSAGE_LOG_FD
+boost_save_IFS=$IFS
+boost_version_req=$1
+IFS=.
+set x $boost_version_req 0 0 0
+IFS=$boost_save_IFS
+shift
+boost_version_req=`expr "$[1]" '*' 100000 + "$[2]" '*' 100 + "$[3]"`
+boost_version_req_string=$[1].$[2].$[3]
+AC_ARG_WITH([boost],
+   [AS_HELP_STRING([--with-boost=DIR],
+                   [prefix of Boost $1 @<:@guess@:>@])])dnl
+AC_ARG_VAR([BOOST_ROOT],[Location of Boost installation])dnl
+# If BOOST_ROOT is set and the user has not provided a value to
+# --with-boost, then treat BOOST_ROOT as if it the user supplied it.
+if test x"$BOOST_ROOT" != x; then
+  if test x"$with_boost" = x; then
+    AC_MSG_NOTICE([Detected BOOST_ROOT; continuing with --with-boost=$BOOST_ROOT])
+    with_boost=$BOOST_ROOT
+  else
+    AC_MSG_NOTICE([Detected BOOST_ROOT=$BOOST_ROOT, but overridden by --with-boost=$with_boost])
+  fi
+fi
+AC_SUBST([DISTCHECK_CONFIGURE_FLAGS],
+         ["$DISTCHECK_CONFIGURE_FLAGS '--with-boost=$with_boost'"])dnl
+boost_save_CPPFLAGS=$CPPFLAGS
+  AC_CACHE_CHECK([for Boost headers version >= $boost_version_req_string],
+    [boost_cv_inc_path],
+    [boost_cv_inc_path=no
+AC_LANG_PUSH([C++])dnl
+m4_pattern_allow([^BOOST_VERSION$])dnl
+    AC_LANG_CONFTEST([AC_LANG_PROGRAM([[#include <boost/version.hpp>
+#if !defined BOOST_VERSION
+# error BOOST_VERSION is not defined
+#elif BOOST_VERSION < $boost_version_req
+# error Boost headers version < $boost_version_req
+#endif
+]])])
+    # If the user provided a value to --with-boost, use it and only it.
+    case $with_boost in #(
+      ''|yes) set x '' /opt/local/include /usr/local/include /opt/include \
+                 /usr/include C:/Boost/include;; #(
+      *)      set x "$with_boost/include" "$with_boost";;
+    esac
+    shift
+    for boost_dir
+    do
+    # Without --layout=system, Boost (or at least some versions) installs
+    # itself in <prefix>/include/boost-<version>.  This inner loop helps to
+    # find headers in such directories.
+    #
+    # Any ${boost_dir}/boost-x_xx directories are searched in reverse version
+    # order followed by ${boost_dir}.  The final '.' is a sentinel for
+    # searching $boost_dir" itself.  Entries are whitespace separated.
+    #
+    # I didn't indent this loop on purpose (to avoid over-indented code)
+    boost_layout_system_search_list=`cd "$boost_dir" 2>/dev/null \
+        && ls -1 | "${GREP}" '^boost-' | sort -rn -t- -k2 \
+        && echo .`
+    for boost_inc in $boost_layout_system_search_list
+    do
+      if test x"$boost_inc" != x.; then
+        boost_inc="$boost_dir/$boost_inc"
+      else
+        boost_inc="$boost_dir" # Uses sentinel in boost_layout_system_search_list
+      fi
+      if test x"$boost_inc" != x; then
+        # We are going to check whether the version of Boost installed
+        # in $boost_inc is usable by running a compilation that
+        # #includes it.  But if we pass a -I/some/path in which Boost
+        # is not installed, the compiler will just skip this -I and
+        # use other locations (either from CPPFLAGS, or from its list
+        # of system include directories).  As a result we would use
+        # header installed on the machine instead of the /some/path
+        # specified by the user.  So in that precise case (trying
+        # $boost_inc), make sure the version.hpp exists.
+        #
+        # Use test -e as there can be symlinks.
+        test -e "$boost_inc/boost/version.hpp" || continue
+        CPPFLAGS="$CPPFLAGS -I$boost_inc"
+      fi
+      AC_COMPILE_IFELSE([], [boost_cv_inc_path=yes], [boost_cv_version=no])
+      if test x"$boost_cv_inc_path" = xyes; then
+        if test x"$boost_inc" != x; then
+          boost_cv_inc_path=$boost_inc
+        fi
+        break 2
+      fi
+    done
+    done
+AC_LANG_POP([C++])dnl
+    ])
+    case $boost_cv_inc_path in #(
+      no)
+        boost_errmsg="cannot find Boost headers version >= $boost_version_req_string"
+        m4_if([$2], [],  [AC_MSG_ERROR([$boost_errmsg])],
+                        [AC_MSG_NOTICE([$boost_errmsg])])
+        $2
+        ;;#(
+      yes)
+        BOOST_CPPFLAGS=
+        ;;#(
+      *)
+        AC_SUBST([BOOST_CPPFLAGS], ["-I$boost_cv_inc_path"])dnl
+        ;;
+    esac
+  if test x"$boost_cv_inc_path" != xno; then
+  AC_DEFINE([HAVE_BOOST], [1],
+            [Defined if the requested minimum BOOST version is satisfied])
+  AC_CACHE_CHECK([for Boost's header version],
+    [boost_cv_lib_version],
+    [m4_pattern_allow([^BOOST_LIB_VERSION$])dnl
+     _BOOST_SED_CPP([[/^boost-lib-version = /{s///;s/[\" ]//g;p;q;}]],
+                    [#include <boost/version.hpp>
+boost-lib-version = BOOST_LIB_VERSION],
+    [boost_cv_lib_version=`cat conftest.i`])])
+    # e.g. "134" for 1_34_1 or "135" for 1_35
+    boost_major_version=`echo "$boost_cv_lib_version" | sed 's/_//;s/_.*//'`
+    case $boost_major_version in #(
+      '' | *[[!0-9]]*)
+        AC_MSG_ERROR([invalid value: boost_major_version='$boost_major_version'])
+        ;;
+    esac
+fi
+CPPFLAGS=$boost_save_CPPFLAGS
+])# BOOST_REQUIRE
+
+
+# BOOST_STATIC()
+# --------------
+# Add the "--enable-static-boost" configure argument. If this argument is given
+# on the command line, static versions of the libraries will be looked up.
+AC_DEFUN([BOOST_STATIC],
+  [AC_ARG_ENABLE([static-boost],
+     [AS_HELP_STRING([--enable-static-boost],
+               [Prefer the static boost libraries over the shared ones [no]])],
+     [enable_static_boost=yes],
+     [enable_static_boost=no])])# BOOST_STATIC
+
+
+# BOOST_FIND_HEADER([HEADER-NAME], [ACTION-IF-NOT-FOUND], [ACTION-IF-FOUND])
+# --------------------------------------------------------------------------
+# Wrapper around AC_CHECK_HEADER for Boost headers.  Useful to check for
+# some parts of the Boost library which are only made of headers and don't
+# require linking (such as Boost.Foreach).
+#
+# Default ACTION-IF-NOT-FOUND: Fail with a fatal error unless Boost couldn't be
+# found in the first place, in which case by default a notice is issued to the
+# user.  Presumably if we haven't died already it's because it's OK to not have
+# Boost, which is why only a notice is issued instead of a hard error.
+#
+# Default ACTION-IF-FOUND: define the preprocessor symbol HAVE_<HEADER-NAME> in
+# case of success # (where HEADER-NAME is written LIKE_THIS, e.g.,
+# HAVE_BOOST_FOREACH_HPP).
+AC_DEFUN([BOOST_FIND_HEADER],
+[AC_REQUIRE([BOOST_REQUIRE])dnl
+if test x"$boost_cv_inc_path" = xno; then
+  m4_default([$2], [AC_MSG_NOTICE([Boost not available, not searching for $1])])
+else
+AC_LANG_PUSH([C++])dnl
+boost_save_CPPFLAGS=$CPPFLAGS
+CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS"
+AC_CHECK_HEADER([$1],
+  [m4_default([$3], [AC_DEFINE(AS_TR_CPP([HAVE_$1]), [1],
+                               [Define to 1 if you have <$1>])])],
+  [m4_default([$2], [AC_MSG_ERROR([cannot find $1])])])
+CPPFLAGS=$boost_save_CPPFLAGS
+AC_LANG_POP([C++])dnl
+fi
+])# BOOST_FIND_HEADER
+
+
+# BOOST_FIND_LIBS([COMPONENT-NAME], [CANDIDATE-LIB-NAMES],
+#                 [PREFERRED-RT-OPT], [HEADER-NAME], [CXX-TEST],
+#                 [CXX-PROLOGUE])
+# --------------------------------------------------------------
+# Look for the Boost library COMPONENT-NAME (e.g., `thread', for
+# libboost_thread) under the possible CANDIDATE-LIB-NAMES (e.g.,
+# "thread_win32 thread").  Check that HEADER-NAME works and check that
+# libboost_LIB-NAME can link with the code CXX-TEST.  The optional
+# argument CXX-PROLOGUE can be used to include some C++ code before
+# the `main' function.
+#
+# Invokes BOOST_FIND_HEADER([HEADER-NAME]) (see above).
+#
+# Boost libraries typically come compiled with several flavors (with different
+# runtime options) so PREFERRED-RT-OPT is the preferred suffix.  A suffix is one
+# or more of the following letters: sgdpn (in that order).  s = static
+# runtime, d = debug build, g = debug/diagnostic runtime, p = STLPort build,
+# n = (unsure) STLPort build without iostreams from STLPort (it looks like `n'
+# must always be used along with `p').  Additionally, PREFERRED-RT-OPT can
+# start with `mt-' to indicate that there is a preference for multi-thread
+# builds.  Some sample values for PREFERRED-RT-OPT: (nothing), mt, d, mt-d, gdp
+# ...  If you want to make sure you have a specific version of Boost
+# (eg, >= 1.33) you *must* invoke BOOST_REQUIRE before this macro.
+AC_DEFUN([BOOST_FIND_LIBS],
+[AC_REQUIRE([BOOST_REQUIRE])dnl
+AC_REQUIRE([_BOOST_FIND_COMPILER_TAG])dnl
+AC_REQUIRE([BOOST_STATIC])dnl
+AC_REQUIRE([_BOOST_GUESS_WHETHER_TO_USE_MT])dnl
+if test x"$boost_cv_inc_path" = xno; then
+  AC_MSG_NOTICE([Boost not available, not searching for the Boost $1 library])
+else
+dnl The else branch is huge and wasn't intended on purpose.
+AC_LANG_PUSH([C++])dnl
+AS_VAR_PUSHDEF([Boost_lib], [boost_cv_lib_$1])dnl
+AS_VAR_PUSHDEF([Boost_lib_LDFLAGS], [boost_cv_lib_$1_LDFLAGS])dnl
+AS_VAR_PUSHDEF([Boost_lib_LDPATH], [boost_cv_lib_$1_LDPATH])dnl
+AS_VAR_PUSHDEF([Boost_lib_LIBS], [boost_cv_lib_$1_LIBS])dnl
+BOOST_FIND_HEADER([$4])
+boost_save_CPPFLAGS=$CPPFLAGS
+CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS"
+AC_CACHE_CHECK([for the Boost $1 library], [Boost_lib],
+               [_BOOST_FIND_LIBS($@)])
+case $Boost_lib in #(
+  (no) _AC_MSG_LOG_CONFTEST
+    AC_MSG_ERROR([cannot find the flags to link with Boost $1])
+    ;;
+esac
+AC_SUBST(AS_TR_CPP([BOOST_$1_LDFLAGS]), [$Boost_lib_LDFLAGS])dnl
+AC_SUBST(AS_TR_CPP([BOOST_$1_LDPATH]), [$Boost_lib_LDPATH])dnl
+AC_SUBST([BOOST_LDPATH], [$Boost_lib_LDPATH])dnl
+AC_SUBST(AS_TR_CPP([BOOST_$1_LIBS]), [$Boost_lib_LIBS])dnl
+CPPFLAGS=$boost_save_CPPFLAGS
+AS_VAR_POPDEF([Boost_lib])dnl
+AS_VAR_POPDEF([Boost_lib_LDFLAGS])dnl
+AS_VAR_POPDEF([Boost_lib_LDPATH])dnl
+AS_VAR_POPDEF([Boost_lib_LIBS])dnl
+AC_LANG_POP([C++])dnl
+fi
+])
+
+
+# BOOST_FIND_LIB([LIB-NAME],
+#                [PREFERRED-RT-OPT], [HEADER-NAME], [CXX-TEST],
+#                [CXX-PROLOGUE])
+# --------------------------------------------------------------
+# Backward compatibility wrapper for BOOST_FIND_LIBS.
+AC_DEFUN([BOOST_FIND_LIB],
+[BOOST_FIND_LIBS([$1], $@)])
+
+
+# _BOOST_FIND_LIBS([LIB-NAME], [CANDIDATE-LIB-NAMES],
+#                 [PREFERRED-RT-OPT], [HEADER-NAME], [CXX-TEST],
+#                 [CXX-PROLOGUE])
+# --------------------------------------------------------------
+# Real implementation of BOOST_FIND_LIBS: rely on these local macros:
+# Boost_lib, Boost_lib_LDFLAGS, Boost_lib_LDPATH, Boost_lib_LIBS
+#
+# The algorithm is as follows: first look for a given library name
+# according to the user's PREFERRED-RT-OPT.  For each library name, we
+# prefer to use the ones that carry the tag (toolset name).  Each
+# library is searched through the various standard paths were Boost is
+# usually installed.  If we can't find the standard variants, we try
+# to enforce -mt (for instance on MacOSX, libboost_thread.dylib
+# doesn't exist but there's -obviously- libboost_thread-mt.dylib).
+AC_DEFUN([_BOOST_FIND_LIBS],
+[Boost_lib=no
+  case "$3" in #(
+    (mt | mt-) boost_mt=-mt; boost_rtopt=;; #(
+    (mt* | mt-*) boost_mt=-mt; boost_rtopt=`expr "X$3" : 'Xmt-*\(.*\)'`;; #(
+    (*) boost_mt=; boost_rtopt=$3;;
+  esac
+  if test $enable_static_boost = yes; then
+    boost_rtopt="s$boost_rtopt"
+  fi
+  # Find the proper debug variant depending on what we've been asked to find.
+  case $boost_rtopt in #(
+    (*d*) boost_rt_d=$boost_rtopt;; #(
+    (*[[sgpn]]*) # Insert the `d' at the right place (in between `sg' and `pn')
+      boost_rt_d=`echo "$boost_rtopt" | sed 's/\(s*g*\)\(p*n*\)/\1\2/'`;; #(
+    (*) boost_rt_d='-d';;
+  esac
+  # If the PREFERRED-RT-OPT are not empty, prepend a `-'.
+  test -n "$boost_rtopt" && boost_rtopt="-$boost_rtopt"
+  $boost_guess_use_mt && boost_mt=-mt
+  # Look for the abs path the static archive.
+  # $libext is computed by Libtool but let's make sure it's non empty.
+  test -z "$libext" &&
+    AC_MSG_ERROR([the libext variable is empty, did you invoke Libtool?])
+  boost_save_ac_objext=$ac_objext
+  # Generate the test file.
+  AC_LANG_CONFTEST([AC_LANG_PROGRAM([#include <$4>
+$6], [$5])])
+dnl Optimization hacks: compiling C++ is slow, especially with Boost.  What
+dnl we're trying to do here is guess the right combination of link flags
+dnl (LIBS / LDFLAGS) to use a given library.  This can take several
+dnl iterations before it succeeds and is thus *very* slow.  So what we do
+dnl instead is that we compile the code first (and thus get an object file,
+dnl typically conftest.o).  Then we try various combinations of link flags
+dnl until we succeed to link conftest.o in an executable.  The problem is
+dnl that the various TRY_LINK / COMPILE_IFELSE macros of Autoconf always
+dnl remove all the temporary files including conftest.o.  So the trick here
+dnl is to temporarily change the value of ac_objext so that conftest.o is
+dnl preserved accross tests.  This is obviously fragile and I will burn in
+dnl hell for not respecting Autoconf's documented interfaces, but in the
+dnl mean time, it optimizes the macro by a factor of 5 to 30.
+dnl Another small optimization: the first argument of AC_COMPILE_IFELSE left
+dnl empty because the test file is generated only once above (before we
+dnl start the for loops).
+  AC_COMPILE_IFELSE([],
+    [ac_objext=do_not_rm_me_plz],
+    [AC_MSG_ERROR([cannot compile a test that uses Boost $1])])
+  ac_objext=$boost_save_ac_objext
+  boost_failed_libs=
+# Don't bother to ident the following nested for loops, only the 2
+# innermost ones matter.
+for boost_lib_ in $2; do
+for boost_tag_ in -$boost_cv_lib_tag ''; do
+for boost_ver_ in -$boost_cv_lib_version ''; do
+for boost_mt_ in $boost_mt -mt ''; do
+for boost_rtopt_ in $boost_rtopt '' -d; do
+  for boost_lib in \
+    boost_$boost_lib_$boost_tag_$boost_mt_$boost_rtopt_$boost_ver_ \
+    boost_$boost_lib_$boost_tag_$boost_rtopt_$boost_ver_ \
+    boost_$boost_lib_$boost_tag_$boost_mt_$boost_ver_ \
+    boost_$boost_lib_$boost_tag_$boost_ver_
+  do
+    # Avoid testing twice the same lib
+    case $boost_failed_libs in #(
+      (*@$boost_lib@*) continue;;
+    esac
+    # If with_boost is empty, we'll search in /lib first, which is not quite
+    # right so instead we'll try to a location based on where the headers are.
+    boost_tmp_lib=$with_boost
+    test x"$with_boost" = x && boost_tmp_lib=${boost_cv_inc_path%/include}
+    for boost_ldpath in "$boost_tmp_lib/lib" '' \
+             /opt/local/lib* /usr/local/lib* /opt/lib* /usr/lib* \
+             "$with_boost" C:/Boost/lib /lib*
+    do
+      # Don't waste time with directories that don't exist.
+      if test x"$boost_ldpath" != x && test ! -e "$boost_ldpath"; then
+        continue
+      fi
+      boost_save_LDFLAGS=$LDFLAGS
+      # Are we looking for a static library?
+      case $boost_ldpath:$boost_rtopt_ in #(
+        (*?*:*s*) # Yes (Non empty boost_ldpath + s in rt opt)
+          Boost_lib_LIBS="$boost_ldpath/lib$boost_lib.$libext"
+          test -e "$Boost_lib_LIBS" || continue;; #(
+        (*) # No: use -lboost_foo to find the shared library.
+          Boost_lib_LIBS="-l$boost_lib";;
+      esac
+      boost_save_LIBS=$LIBS
+      LIBS="$Boost_lib_LIBS $LIBS"
+      test x"$boost_ldpath" != x && LDFLAGS="$LDFLAGS -L$boost_ldpath"
+dnl First argument of AC_LINK_IFELSE left empty because the test file is
+dnl generated only once above (before we start the for loops).
+      _BOOST_AC_LINK_IFELSE([],
+                            [Boost_lib=yes], [Boost_lib=no])
+      ac_objext=$boost_save_ac_objext
+      LDFLAGS=$boost_save_LDFLAGS
+      LIBS=$boost_save_LIBS
+      if test x"$Boost_lib" = xyes; then
+        test x"$boost_ldpath" != x &&
+          Boost_lib_LDFLAGS="-L$boost_ldpath $boost_cv_rpath_link_ldflag$boost_ldpath"
+        Boost_lib_LDPATH="$boost_ldpath"
+        break 7
+      else
+        boost_failed_libs="$boost_failed_libs@$boost_lib@"
+      fi
+    done
+  done
+done
+done
+done
+done
+done # boost_lib_
+rm -f conftest.$ac_objext
+])
+
+
+
+# --------------------------------------- #
+# Checks for the various Boost libraries. #
+# --------------------------------------- #
+
+# List of boost libraries: http://www.boost.org/libs/libraries.htm
+# The page http://beta.boost.org/doc/libs is useful: it gives the first release
+# version of each library (among other things).
+
+# BOOST_DEFUN(LIBRARY, CODE)
+# --------------------------
+# Define BOOST_<LIBRARY-UPPERCASE> as a macro that runs CODE.
+#
+# Use indir to avoid the warning on underquoted macro name given to AC_DEFUN.
+m4_define([BOOST_DEFUN],
+[m4_indir([AC_DEFUN],
+          m4_toupper([BOOST_$1]),
+[m4_pushdef([BOOST_Library], [$1])dnl
+$2
+m4_popdef([BOOST_Library])dnl
+])
+])
+
+# BOOST_ARRAY()
+# -------------
+# Look for Boost.Array
+BOOST_DEFUN([Array],
+[BOOST_FIND_HEADER([boost/array.hpp])])
+
+
+# BOOST_ASIO()
+# ------------
+# Look for Boost.Asio (new in Boost 1.35).
+BOOST_DEFUN([Asio],
+[AC_REQUIRE([BOOST_SYSTEM])dnl
+BOOST_FIND_HEADER([boost/asio.hpp])])
+
+
+# BOOST_BIND()
+# ------------
+# Look for Boost.Bind.
+BOOST_DEFUN([Bind],
+[BOOST_FIND_HEADER([boost/bind.hpp])])
+
+
+# BOOST_CHRONO()
+# --------------
+# Look for Boost.Chrono.
+BOOST_DEFUN([Chrono],
+[# Do we have to check for Boost.System?  This link-time dependency was
+# added as of 1.35.0.  If we have a version <1.35, we must not attempt to
+# find Boost.System as it didn't exist by then.
+if test $boost_major_version -ge 135; then
+  BOOST_SYSTEM([$1])
+fi # end of the Boost.System check.
+boost_filesystem_save_LIBS=$LIBS
+boost_filesystem_save_LDFLAGS=$LDFLAGS
+m4_pattern_allow([^BOOST_SYSTEM_(LIBS|LDFLAGS)$])dnl
+LIBS="$LIBS $BOOST_SYSTEM_LIBS"
+LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS"
+BOOST_FIND_LIB([chrono], [$1],
+                [boost/chrono.hpp],
+                [boost::chrono::thread_clock d;])
+if test $enable_static_boost = yes && test $boost_major_version -ge 135; then
+  BOOST_CHRONO_LIBS="$BOOST_CHRONO_LIBS $BOOST_SYSTEM_LIBS"
+fi
+LIBS=$boost_filesystem_save_LIBS
+LDFLAGS=$boost_filesystem_save_LDFLAGS
+])# BOOST_CHRONO
+
+
+# BOOST_CONTEXT([PREFERRED-RT-OPT])
+# -----------------------------------
+# Look for Boost.Context.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.  This library was introduced in Boost
+# 1.51.0
+BOOST_DEFUN([Context],
+[BOOST_FIND_LIB([context], [$1],
+                [boost/context/all.hpp],[[
+// creates a stack
+void * stack_pointer = new void*[4096];
+std::size_t const size = sizeof(void*[4096]);
+
+// context fc uses f() as context function
+// fcontext_t is placed on top of context stack
+// a pointer to fcontext_t is returned
+fc = ctx::make_fcontext(stack_pointer, size, f);
+return ctx::jump_fcontext(&fcm, fc, 3) == 6;]],[dnl
+namespace ctx = boost::context;
+// context
+static ctx::fcontext_t fcm, *fc;
+// context-function
+static void f(intptr_t i) {
+    ctx::jump_fcontext(fc, &fcm, i * 2);
+}])
+])# BOOST_CONTEXT
+
+
+# BOOST_CONVERSION()
+# ------------------
+# Look for Boost.Conversion (cast / lexical_cast)
+BOOST_DEFUN([Conversion],
+[BOOST_FIND_HEADER([boost/cast.hpp])
+BOOST_FIND_HEADER([boost/lexical_cast.hpp])
+])# BOOST_CONVERSION
+
+
+# BOOST_COROUTINE([PREFERRED-RT-OPT])
+# -----------------------------------
+# Look for Boost.Coroutine.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.  This library was introduced in Boost
+# 1.53.0
+BOOST_DEFUN([Coroutine],
+[
+boost_coroutine_save_LIBS=$LIBS
+boost_coroutine_save_LDFLAGS=$LDFLAGS
+# Link-time dependency from coroutine to context
+BOOST_CONTEXT([$1])
+# Starting from Boost 1.55 a dependency on Boost.System is added
+if test $boost_major_version -ge 155; then
+  BOOST_SYSTEM([$1])
+fi
+m4_pattern_allow([^BOOST_(CONTEXT|SYSTEM)_(LIBS|LDFLAGS)])
+LIBS="$LIBS $BOOST_CONTEXT_LIBS $BOOST_SYSTEM_LIBS"
+LDFLAGS="$LDFLAGS $BOOST_CONTEXT_LDFLAGS"
+
+BOOST_FIND_LIB([coroutine], [$1],
+                [boost/coroutine/coroutine.hpp],
+                [boost::coroutines::coroutine< int(int) > coro; coro.empty();])
+
+# Link-time dependency from coroutine to context, existed only in 1.53, in 1.54
+# coroutine doesn't use context from its headers but from its library.
+if test $boost_major_version -eq 153 || test $enable_static_boost = yes && test $boost_major_version -ge 154; then
+  BOOST_COROUTINE_LIBS="$BOOST_COROUTINE_LIBS $BOOST_CONTEXT_LIBS"
+  BOOST_COROUTINE_LDFLAGS="$BOOST_COROUTINE_LDFLAGS $BOOST_CONTEXT_LDFLAGS"
+fi
+if test $enable_static_boost = yes && test $boost_major_version -ge 155; then
+  BOOST_COROUTINE_LIBS="$BOOST_COROUTINE_LIBS $BOOST_SYSTEM_LIBS"
+  BOOST_COROUTINE_LDFLAGS="$BOOST_COROUTINE_LDFLAGS $BOOST_SYSTEM_LDFLAGS"
+fi
+LIBS=$boost_coroutine_save_LIBS
+LDFLAGS=$boost_coroutine_save_LDFLAGS
+])# BOOST_COROUTINE
+
+
+# BOOST_CRC()
+# -----------
+# Look for Boost.CRC
+BOOST_DEFUN([CRC],
+[BOOST_FIND_HEADER([boost/crc.hpp])
+])# BOOST_CRC
+
+
+# BOOST_DATE_TIME([PREFERRED-RT-OPT])
+# -----------------------------------
+# Look for Boost.Date_Time.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Date_Time],
+[BOOST_FIND_LIB([date_time], [$1],
+                [boost/date_time/posix_time/posix_time.hpp],
+                [boost::posix_time::ptime t;])
+])# BOOST_DATE_TIME
+
+
+# BOOST_FILESYSTEM([PREFERRED-RT-OPT])
+# ------------------------------------
+# Look for Boost.Filesystem.  For the documentation of PREFERRED-RT-OPT, see
+# the documentation of BOOST_FIND_LIB above.
+# Do not check for boost/filesystem.hpp because this file was introduced in
+# 1.34.
+BOOST_DEFUN([Filesystem],
+[# Do we have to check for Boost.System?  This link-time dependency was
+# added as of 1.35.0.  If we have a version <1.35, we must not attempt to
+# find Boost.System as it didn't exist by then.
+if test $boost_major_version -ge 135; then
+  BOOST_SYSTEM([$1])
+fi # end of the Boost.System check.
+boost_filesystem_save_LIBS=$LIBS
+boost_filesystem_save_LDFLAGS=$LDFLAGS
+m4_pattern_allow([^BOOST_SYSTEM_(LIBS|LDFLAGS)$])dnl
+LIBS="$LIBS $BOOST_SYSTEM_LIBS"
+LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS"
+BOOST_FIND_LIB([filesystem], [$1],
+                [boost/filesystem/path.hpp], [boost::filesystem::path p;])
+if test $enable_static_boost = yes && test $boost_major_version -ge 135; then
+  BOOST_FILESYSTEM_LIBS="$BOOST_FILESYSTEM_LIBS $BOOST_SYSTEM_LIBS"
+fi
+LIBS=$boost_filesystem_save_LIBS
+LDFLAGS=$boost_filesystem_save_LDFLAGS
+])# BOOST_FILESYSTEM
+
+
+# BOOST_FLYWEIGHT()
+# -----------------
+# Look for Boost.Flyweight.
+BOOST_DEFUN([Flyweight],
+[dnl There's a hidden dependency on pthreads.
+AC_REQUIRE([_BOOST_PTHREAD_FLAG])dnl
+BOOST_FIND_HEADER([boost/flyweight.hpp])
+AC_SUBST([BOOST_FLYWEIGHT_LIBS], [$boost_cv_pthread_flag])
+])
+
+
+# BOOST_FOREACH()
+# ---------------
+# Look for Boost.Foreach.
+BOOST_DEFUN([Foreach],
+[BOOST_FIND_HEADER([boost/foreach.hpp])])
+
+
+# BOOST_FORMAT()
+# --------------
+# Look for Boost.Format.
+# Note: we can't check for boost/format/format_fwd.hpp because the header isn't
+# standalone.  It can't be compiled because it triggers the following error:
+# boost/format/detail/config_macros.hpp:88: error: 'locale' in namespace 'std'
+#                                                  does not name a type
+BOOST_DEFUN([Format],
+[BOOST_FIND_HEADER([boost/format.hpp])])
+
+
+# BOOST_FUNCTION()
+# ----------------
+# Look for Boost.Function
+BOOST_DEFUN([Function],
+[BOOST_FIND_HEADER([boost/function.hpp])])
+
+
+# BOOST_GEOMETRY()
+# ----------------
+# Look for Boost.Geometry (new since 1.47.0).
+BOOST_DEFUN([Geometry],
+[BOOST_FIND_HEADER([boost/geometry.hpp])
+])# BOOST_GEOMETRY
+
+
+# BOOST_GRAPH([PREFERRED-RT-OPT])
+# -------------------------------
+# Look for Boost.Graphs.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Graph],
+[BOOST_FIND_LIB([graph], [$1],
+                [boost/graph/adjacency_list.hpp], [boost::adjacency_list<> g;])
+])# BOOST_GRAPH
+
+
+# BOOST_IOSTREAMS([PREFERRED-RT-OPT])
+# -----------------------------------
+# Look for Boost.IOStreams.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([IOStreams],
+[BOOST_FIND_LIB([iostreams], [$1],
+                [boost/iostreams/device/file_descriptor.hpp],
+                [boost::iostreams::file_descriptor fd; fd.close();])
+])# BOOST_IOSTREAMS
+
+
+# BOOST_HASH()
+# ------------
+# Look for Boost.Functional/Hash
+BOOST_DEFUN([Hash],
+[BOOST_FIND_HEADER([boost/functional/hash.hpp])])
+
+
+# BOOST_LAMBDA()
+# --------------
+# Look for Boost.Lambda
+BOOST_DEFUN([Lambda],
+[BOOST_FIND_HEADER([boost/lambda/lambda.hpp])])
+
+
+# BOOST_LOCALE()
+# --------------
+# Look for Boost.Locale
+BOOST_DEFUN([Locale],
+[BOOST_FIND_LIB([locale], [$1],
+    [boost/locale.hpp],
+    [[boost::locale::generator gen; std::locale::global(gen(""));]])
+])# BOOST_LOCALE
+
+# BOOST_LOG([PREFERRED-RT-OPT])
+# -----------------------------
+# Look for Boost.Log.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Log],
+[BOOST_FIND_LIB([log], [$1],
+    [boost/log/core/core.hpp],
+    [boost::log::attribute a; a.get_value();])
+])# BOOST_LOG
+
+
+# BOOST_LOG_SETUP([PREFERRED-RT-OPT])
+# -----------------------------------
+# Look for Boost.Log.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Log_Setup],
+[AC_REQUIRE([BOOST_LOG])dnl
+BOOST_FIND_LIB([log_setup], [$1],
+    [boost/log/utility/setup/from_settings.hpp],
+    [boost::log::basic_settings<char> bs; bs.empty();])
+])# BOOST_LOG_SETUP
+
+
+# BOOST_MATH()
+# ------------
+# Look for Boost.Math
+# TODO: This library isn't header-only but it comes in multiple different
+# flavors that don't play well with BOOST_FIND_LIB (e.g, libboost_math_c99,
+# libboost_math_c99f, libboost_math_c99l, libboost_math_tr1,
+# libboost_math_tr1f, libboost_math_tr1l).  This macro must be fixed to do the
+# right thing anyway.
+BOOST_DEFUN([Math],
+[BOOST_FIND_HEADER([boost/math/special_functions.hpp])])
+
+
+# BOOST_MPI([PREFERRED-RT-OPT])
+# -------------------------------
+# Look for Boost MPI.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.  Uses MPICXX variable if it is
+# set, otherwise tries CXX
+#
+BOOST_DEFUN([MPI],
+[boost_save_CXX=${CXX}
+boost_save_CXXCPP=${CXXCPP}
+if test x"${MPICXX}" != x; then
+  CXX=${MPICXX}
+  CXXCPP="${MPICXX} -E"
+fi
+BOOST_FIND_LIB([mpi], [$1],
+               [boost/mpi.hpp],
+               [int argc = 0;
+                char **argv = 0;
+                boost::mpi::environment env(argc,argv);])
+CXX=${boost_save_CXX}
+CXXCPP=${boost_save_CXXCPP}
+])# BOOST_MPI
+
+
+# BOOST_MULTIARRAY()
+# ------------------
+# Look for Boost.MultiArray
+BOOST_DEFUN([MultiArray],
+[BOOST_FIND_HEADER([boost/multi_array.hpp])])
+
+
+# BOOST_NUMERIC_UBLAS()
+# --------------------------
+# Look for Boost.NumericUblas (Basic Linear Algebra)
+BOOST_DEFUN([Numeric_Ublas],
+[BOOST_FIND_HEADER([boost/numeric/ublas/vector.hpp])
+])# BOOST_NUMERIC_UBLAS
+
+
+# BOOST_NUMERIC_CONVERSION()
+# --------------------------
+# Look for Boost.NumericConversion (policy-based numeric conversion)
+BOOST_DEFUN([Numeric_Conversion],
+[BOOST_FIND_HEADER([boost/numeric/conversion/converter.hpp])
+])# BOOST_NUMERIC_CONVERSION
+
+
+# BOOST_OPTIONAL()
+# ----------------
+# Look for Boost.Optional
+BOOST_DEFUN([Optional],
+[BOOST_FIND_HEADER([boost/optional.hpp])])
+
+
+# BOOST_PREPROCESSOR()
+# --------------------
+# Look for Boost.Preprocessor
+BOOST_DEFUN([Preprocessor],
+[BOOST_FIND_HEADER([boost/preprocessor/repeat.hpp])])
+
+
+# BOOST_RANGE()
+# --------------------
+# Look for Boost.Range
+BOOST_DEFUN([Range],
+[BOOST_FIND_HEADER([boost/range/adaptors.hpp])])
+
+# BOOST_UNORDERED()
+# -----------------
+# Look for Boost.Unordered
+BOOST_DEFUN([Unordered],
+[BOOST_FIND_HEADER([boost/unordered_map.hpp])])
+
+
+# BOOST_UUID()
+# ------------
+# Look for Boost.Uuid
+BOOST_DEFUN([Uuid],
+[BOOST_FIND_HEADER([boost/uuid/uuid.hpp])])
+
+
+# BOOST_PROGRAM_OPTIONS([PREFERRED-RT-OPT])
+# -----------------------------------------
+# Look for Boost.Program_options.  For the documentation of PREFERRED-RT-OPT,
+# see the documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Program_Options],
+[BOOST_FIND_LIB([program_options], [$1],
+                [boost/program_options.hpp],
+                [boost::program_options::options_description d("test");])
+])# BOOST_PROGRAM_OPTIONS
+
+
+
+# _BOOST_PYTHON_CONFIG(VARIABLE, FLAG)
+# ------------------------------------
+# Save VARIABLE, and define it via `python-config --FLAG`.
+# Substitute BOOST_PYTHON_VARIABLE.
+m4_define([_BOOST_PYTHON_CONFIG],
+[AC_SUBST([BOOST_PYTHON_$1],
+          [`python-config --$2 2>/dev/null`])dnl
+boost_python_save_$1=$$1
+$1="$$1 $BOOST_PYTHON_$1"])
+
+
+# BOOST_PYTHON([PREFERRED-RT-OPT])
+# --------------------------------
+# Look for Boost.Python.  For the documentation of PREFERRED-RT-OPT,
+# see the documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Python],
+[_BOOST_PYTHON_CONFIG([CPPFLAGS], [includes])
+_BOOST_PYTHON_CONFIG([LDFLAGS],   [ldflags])
+_BOOST_PYTHON_CONFIG([LIBS],      [libs])
+m4_pattern_allow([^BOOST_PYTHON_MODULE$])dnl
+BOOST_FIND_LIBS([python], [python python3], [$1],
+                [boost/python.hpp],
+                [], [BOOST_PYTHON_MODULE(empty) {}])
+CPPFLAGS=$boost_python_save_CPPFLAGS
+LDFLAGS=$boost_python_save_LDFLAGS
+LIBS=$boost_python_save_LIBS
+])# BOOST_PYTHON
+
+
+# BOOST_REF()
+# -----------
+# Look for Boost.Ref
+BOOST_DEFUN([Ref],
+[BOOST_FIND_HEADER([boost/ref.hpp])])
+
+
+# BOOST_REGEX([PREFERRED-RT-OPT])
+# -------------------------------
+# Look for Boost.Regex.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Regex],
+[BOOST_FIND_LIB([regex], [$1],
+                [boost/regex.hpp],
+                [boost::regex exp("*"); boost::regex_match("foo", exp);])
+])# BOOST_REGEX
+
+
+# BOOST_SERIALIZATION([PREFERRED-RT-OPT])
+# ---------------------------------------
+# Look for Boost.Serialization.  For the documentation of PREFERRED-RT-OPT, see
+# the documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Serialization],
+[BOOST_FIND_LIB([serialization], [$1],
+                [boost/archive/text_oarchive.hpp],
+                [std::ostream* o = 0; // Cheap way to get an ostream...
+                boost::archive::text_oarchive t(*o);])
+])# BOOST_SERIALIZATION
+
+
+# BOOST_SIGNALS([PREFERRED-RT-OPT])
+# ---------------------------------
+# Look for Boost.Signals.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Signals],
+[BOOST_FIND_LIB([signals], [$1],
+                [boost/signal.hpp],
+                [boost::signal<void ()> s;])
+])# BOOST_SIGNALS
+
+
+# BOOST_SIGNALS2()
+# ----------------
+# Look for Boost.Signals2 (new since 1.39.0).
+BOOST_DEFUN([Signals2],
+[BOOST_FIND_HEADER([boost/signals2.hpp])
+])# BOOST_SIGNALS2
+
+
+# BOOST_SMART_PTR()
+# -----------------
+# Look for Boost.SmartPtr
+BOOST_DEFUN([Smart_Ptr],
+[BOOST_FIND_HEADER([boost/scoped_ptr.hpp])
+BOOST_FIND_HEADER([boost/shared_ptr.hpp])
+])
+
+
+# BOOST_STATICASSERT()
+# --------------------
+# Look for Boost.StaticAssert
+BOOST_DEFUN([StaticAssert],
+[BOOST_FIND_HEADER([boost/static_assert.hpp])])
+
+
+# BOOST_STRING_ALGO()
+# -------------------
+# Look for Boost.StringAlgo
+BOOST_DEFUN([String_Algo],
+[BOOST_FIND_HEADER([boost/algorithm/string.hpp])
+])
+
+
+# BOOST_SYSTEM([PREFERRED-RT-OPT])
+# --------------------------------
+# Look for Boost.System.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.  This library was introduced in Boost
+# 1.35.0.
+BOOST_DEFUN([System],
+[BOOST_FIND_LIB([system], [$1],
+                [boost/system/error_code.hpp],
+                [boost::system::error_code e; e.clear();])
+])# BOOST_SYSTEM
+
+
+# BOOST_TEST([PREFERRED-RT-OPT])
+# ------------------------------
+# Look for Boost.Test.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Test],
+[m4_pattern_allow([^BOOST_CHECK$])dnl
+BOOST_FIND_LIB([unit_test_framework], [$1],
+               [boost/test/unit_test.hpp], [BOOST_CHECK(2 == 2);],
+               [using boost::unit_test::test_suite;
+               test_suite* init_unit_test_suite(int argc, char ** argv)
+               { return NULL; }])
+])# BOOST_TEST
+
+
+# BOOST_THREAD([PREFERRED-RT-OPT])
+# ---------------------------------
+# Look for Boost.Thread.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Thread],
+[dnl Having the pthread flag is required at least on GCC3 where
+dnl boost/thread.hpp would complain if we try to compile without
+dnl -pthread on GNU/Linux.
+AC_REQUIRE([_BOOST_PTHREAD_FLAG])dnl
+boost_thread_save_LIBS=$LIBS
+boost_thread_save_LDFLAGS=$LDFLAGS
+boost_thread_save_CPPFLAGS=$CPPFLAGS
+# Link-time dependency from thread to system was added as of 1.49.0.
+if test $boost_major_version -ge 149; then
+BOOST_SYSTEM([$1])
+fi # end of the Boost.System check.
+m4_pattern_allow([^BOOST_SYSTEM_(LIBS|LDFLAGS)$])dnl
+LIBS="$LIBS $BOOST_SYSTEM_LIBS $boost_cv_pthread_flag"
+LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS"
+CPPFLAGS="$CPPFLAGS $boost_cv_pthread_flag"
+
+# When compiling for the Windows platform, the threads library is named
+# differently.
+case $host_os in
+  (*mingw*) boost_thread_lib_ext=_win32;;
+esac
+BOOST_FIND_LIBS([thread], [thread$boost_thread_lib_ext],
+                [$1],
+                [boost/thread.hpp], [boost::thread t; boost::mutex m;])
+
+BOOST_THREAD_LIBS="$BOOST_THREAD_LIBS $BOOST_SYSTEM_LIBS $boost_cv_pthread_flag"
+BOOST_THREAD_LDFLAGS="$BOOST_SYSTEM_LDFLAGS"
+BOOST_CPPFLAGS="$BOOST_CPPFLAGS $boost_cv_pthread_flag"
+LIBS=$boost_thread_save_LIBS
+LDFLAGS=$boost_thread_save_LDFLAGS
+CPPFLAGS=$boost_thread_save_CPPFLAGS
+])# BOOST_THREAD
+
+AU_ALIAS([BOOST_THREADS], [BOOST_THREAD])
+
+
+# BOOST_TOKENIZER()
+# -----------------
+# Look for Boost.Tokenizer
+BOOST_DEFUN([Tokenizer],
+[BOOST_FIND_HEADER([boost/tokenizer.hpp])])
+
+
+# BOOST_TRIBOOL()
+# ---------------
+# Look for Boost.Tribool
+BOOST_DEFUN([Tribool],
+[BOOST_FIND_HEADER([boost/logic/tribool_fwd.hpp])
+BOOST_FIND_HEADER([boost/logic/tribool.hpp])
+])
+
+
+# BOOST_TUPLE()
+# -------------
+# Look for Boost.Tuple
+BOOST_DEFUN([Tuple],
+[BOOST_FIND_HEADER([boost/tuple/tuple.hpp])])
+
+
+# BOOST_TYPETRAITS()
+# --------------------
+# Look for Boost.TypeTraits
+BOOST_DEFUN([TypeTraits],
+[BOOST_FIND_HEADER([boost/type_traits.hpp])])
+
+
+# BOOST_UTILITY()
+# ---------------
+# Look for Boost.Utility (noncopyable, result_of, base-from-member idiom,
+# etc.)
+BOOST_DEFUN([Utility],
+[BOOST_FIND_HEADER([boost/utility.hpp])])
+
+
+# BOOST_VARIANT()
+# ---------------
+# Look for Boost.Variant.
+BOOST_DEFUN([Variant],
+[BOOST_FIND_HEADER([boost/variant/variant_fwd.hpp])
+BOOST_FIND_HEADER([boost/variant.hpp])])
+
+
+# BOOST_POINTER_CONTAINER()
+# ------------------------
+# Look for Boost.PointerContainer
+BOOST_DEFUN([Pointer_Container],
+[BOOST_FIND_HEADER([boost/ptr_container/ptr_deque.hpp])
+BOOST_FIND_HEADER([boost/ptr_container/ptr_list.hpp])
+BOOST_FIND_HEADER([boost/ptr_container/ptr_vector.hpp])
+BOOST_FIND_HEADER([boost/ptr_container/ptr_array.hpp])
+BOOST_FIND_HEADER([boost/ptr_container/ptr_set.hpp])
+BOOST_FIND_HEADER([boost/ptr_container/ptr_map.hpp])
+])# BOOST_POINTER_CONTAINER
+
+
+# BOOST_WAVE([PREFERRED-RT-OPT])
+# ------------------------------
+# NOTE: If you intend to use Wave/Spirit with thread support, make sure you
+# call BOOST_THREAD first.
+# Look for Boost.Wave.  For the documentation of PREFERRED-RT-OPT, see the
+# documentation of BOOST_FIND_LIB above.
+BOOST_DEFUN([Wave],
+[AC_REQUIRE([BOOST_FILESYSTEM])dnl
+AC_REQUIRE([BOOST_DATE_TIME])dnl
+boost_wave_save_LIBS=$LIBS
+boost_wave_save_LDFLAGS=$LDFLAGS
+m4_pattern_allow([^BOOST_((FILE)?SYSTEM|DATE_TIME|THREAD)_(LIBS|LDFLAGS)$])dnl
+LIBS="$LIBS $BOOST_SYSTEM_LIBS $BOOST_FILESYSTEM_LIBS $BOOST_DATE_TIME_LIBS \
+$BOOST_THREAD_LIBS"
+LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS $BOOST_FILESYSTEM_LDFLAGS \
+$BOOST_DATE_TIME_LDFLAGS $BOOST_THREAD_LDFLAGS"
+BOOST_FIND_LIB([wave], [$1],
+                [boost/wave.hpp],
+                [boost::wave::token_id id; get_token_name(id);])
+LIBS=$boost_wave_save_LIBS
+LDFLAGS=$boost_wave_save_LDFLAGS
+])# BOOST_WAVE
+
+
+# BOOST_XPRESSIVE()
+# -----------------
+# Look for Boost.Xpressive (new since 1.36.0).
+BOOST_DEFUN([Xpressive],
+[BOOST_FIND_HEADER([boost/xpressive/xpressive.hpp])])
+
+
+# ----------------- #
+# Internal helpers. #
+# ----------------- #
+
+
+# _BOOST_PTHREAD_FLAG()
+# ---------------------
+# Internal helper for BOOST_THREAD.  Computes boost_cv_pthread_flag
+# which must be used in CPPFLAGS and LIBS.
+#
+# Yes, we *need* to put the -pthread thing in CPPFLAGS because with GCC3,
+# boost/thread.hpp will trigger a #error if -pthread isn't used:
+#   boost/config/requires_threads.hpp:47:5: #error "Compiler threading support
+#   is not turned on. Please set the correct command line options for
+#   threading: -pthread (Linux), -pthreads (Solaris) or -mthreads (Mingw32)"
+#
+# Based on ACX_PTHREAD: http://autoconf-archive.cryp.to/acx_pthread.html
+AC_DEFUN([_BOOST_PTHREAD_FLAG],
+[AC_REQUIRE([AC_PROG_CXX])dnl
+AC_REQUIRE([AC_CANONICAL_HOST])dnl
+AC_LANG_PUSH([C++])dnl
+AC_CACHE_CHECK([for the flags needed to use pthreads], [boost_cv_pthread_flag],
+[ boost_cv_pthread_flag=
+  # The ordering *is* (sometimes) important.  Some notes on the
+  # individual items follow:
+  # (none): in case threads are in libc; should be tried before -Kthread and
+  #       other compiler flags to prevent continual compiler warnings
+  # -lpthreads: AIX (must check this before -lpthread)
+  # -Kthread: Sequent (threads in libc, but -Kthread needed for pthread.h)
+  # -kthread: FreeBSD kernel threads (preferred to -pthread since SMP-able)
+  # -llthread: LinuxThreads port on FreeBSD (also preferred to -pthread)
+  # -pthread: GNU Linux/GCC (kernel threads), BSD/GCC (userland threads)
+  # -pthreads: Solaris/GCC
+  # -mthreads: MinGW32/GCC, Lynx/GCC
+  # -mt: Sun Workshop C (may only link SunOS threads [-lthread], but it
+  #      doesn't hurt to check since this sometimes defines pthreads too;
+  #      also defines -D_REENTRANT)
+  #      ... -mt is also the pthreads flag for HP/aCC
+  # -lpthread: GNU Linux, etc.
+  # --thread-safe: KAI C++
+  case $host_os in #(
+    *solaris*)
+      # On Solaris (at least, for some versions), libc contains stubbed
+      # (non-functional) versions of the pthreads routines, so link-based
+      # tests will erroneously succeed.  (We need to link with -pthreads/-mt/
+      # -lpthread.)  (The stubs are missing pthread_cleanup_push, or rather
+      # a function called by this macro, so we could check for that, but
+      # who knows whether they'll stub that too in a future libc.)  So,
+      # we'll just look for -pthreads and -lpthread first:
+      boost_pthread_flags="-pthreads -lpthread -mt -pthread";; #(
+    *)
+      boost_pthread_flags="-lpthreads -Kthread -kthread -llthread -pthread \
+                           -pthreads -mthreads -lpthread --thread-safe -mt";;
+  esac
+  # Generate the test file.
+  AC_LANG_CONFTEST([AC_LANG_PROGRAM([#include <pthread.h>],
+    [pthread_t th; pthread_join(th, 0);
+    pthread_attr_init(0); pthread_cleanup_push(0, 0);
+    pthread_create(0,0,0,0); pthread_cleanup_pop(0);])])
+  for boost_pthread_flag in '' $boost_pthread_flags; do
+    boost_pthread_ok=false
+dnl Re-use the test file already generated.
+    boost_pthreads__save_LIBS=$LIBS
+    LIBS="$LIBS $boost_pthread_flag"
+    AC_LINK_IFELSE([],
+      [if grep ".*$boost_pthread_flag" conftest.err; then
+         echo "This flag seems to have triggered warnings" >&AS_MESSAGE_LOG_FD
+       else
+         boost_pthread_ok=:; boost_cv_pthread_flag=$boost_pthread_flag
+       fi])
+    LIBS=$boost_pthreads__save_LIBS
+    $boost_pthread_ok && break
+  done
+])
+AC_LANG_POP([C++])dnl
+])# _BOOST_PTHREAD_FLAG
+
+
+# _BOOST_gcc_test(MAJOR, MINOR)
+# -----------------------------
+# Internal helper for _BOOST_FIND_COMPILER_TAG.
+m4_define([_BOOST_gcc_test],
+["defined __GNUC__ && __GNUC__ == $1 && __GNUC_MINOR__ == $2 && !defined __ICC @ gcc$1$2"])dnl
+
+# _BOOST_mingw_test(MAJOR, MINOR)
+# -----------------------------
+# Internal helper for _BOOST_FIND_COMPILER_TAG.
+m4_define([_BOOST_mingw_test],
+["defined __GNUC__ && __GNUC__ == $1 && __GNUC_MINOR__ == $2 && !defined __ICC && \
+  (defined WIN32 || defined WINNT || defined _WIN32 || defined __WIN32 \
+         || defined __WIN32__ || defined __WINNT || defined __WINNT__) @ mgw$1$2"])dnl
+
+
+# _BOOST_FIND_COMPILER_TAG()
+# --------------------------
+# Internal.  When Boost is installed without --layout=system, each library
+# filename will hold a suffix that encodes the compiler used during the
+# build.  The Boost build system seems to call this a `tag'.
+AC_DEFUN([_BOOST_FIND_COMPILER_TAG],
+[AC_REQUIRE([AC_PROG_CXX])dnl
+AC_REQUIRE([AC_CANONICAL_HOST])dnl
+AC_CACHE_CHECK([for the toolset name used by Boost for $CXX],
+               [boost_cv_lib_tag],
+[boost_cv_lib_tag=unknown
+if test x$boost_cv_inc_path != xno; then
+  AC_LANG_PUSH([C++])dnl
+  # The following tests are mostly inspired by boost/config/auto_link.hpp
+  # The list is sorted to most recent/common to oldest compiler (in order
+  # to increase the likelihood of finding the right compiler with the
+  # least number of compilation attempt).
+  # Beware that some tests are sensible to the order (for instance, we must
+  # look for MinGW before looking for GCC3).
+  # I used one compilation test per compiler with a #error to recognize
+  # each compiler so that it works even when cross-compiling (let me know
+  # if you know a better approach).
+  # Known missing tags (known from Boost's tools/build/v2/tools/common.jam):
+  #   como, edg, kcc, bck, mp, sw, tru, xlc
+  # I'm not sure about my test for `il' (be careful: Intel's ICC pre-defines
+  # the same defines as GCC's).
+  for i in \
+    _BOOST_mingw_test(5, 0) \
+    _BOOST_gcc_test(5, 0) \
+    _BOOST_mingw_test(4, 10) \
+    _BOOST_gcc_test(4, 10) \
+    _BOOST_mingw_test(4, 9) \
+    _BOOST_gcc_test(4, 9) \
+    _BOOST_mingw_test(4, 8) \
+    _BOOST_gcc_test(4, 8) \
+    _BOOST_mingw_test(4, 7) \
+    _BOOST_gcc_test(4, 7) \
+    _BOOST_mingw_test(4, 6) \
+    _BOOST_gcc_test(4, 6) \
+    _BOOST_mingw_test(4, 5) \
+    _BOOST_gcc_test(4, 5) \
+    _BOOST_mingw_test(4, 4) \
+    _BOOST_gcc_test(4, 4) \
+    _BOOST_mingw_test(4, 3) \
+    _BOOST_gcc_test(4, 3) \
+    _BOOST_mingw_test(4, 2) \
+    _BOOST_gcc_test(4, 2) \
+    _BOOST_mingw_test(4, 1) \
+    _BOOST_gcc_test(4, 1) \
+    _BOOST_mingw_test(4, 0) \
+    _BOOST_gcc_test(4, 0) \
+    "defined __GNUC__ && __GNUC__ == 3 && !defined __ICC \
+     && (defined WIN32 || defined WINNT || defined _WIN32 || defined __WIN32 \
+         || defined __WIN32__ || defined __WINNT || defined __WINNT__) @ mgw" \
+    _BOOST_gcc_test(3, 4) \
+    _BOOST_gcc_test(3, 3) \
+    "defined _MSC_VER && _MSC_VER >= 1500 @ vc90" \
+    "defined _MSC_VER && _MSC_VER == 1400 @ vc80" \
+    _BOOST_gcc_test(3, 2) \
+    "defined _MSC_VER && _MSC_VER == 1310 @ vc71" \
+    _BOOST_gcc_test(3, 1) \
+    _BOOST_gcc_test(3, 0) \
+    "defined __BORLANDC__ @ bcb" \
+    "defined __ICC && (defined __unix || defined __unix__) @ il" \
+    "defined __ICL @ iw" \
+    "defined _MSC_VER && _MSC_VER == 1300 @ vc7" \
+    _BOOST_gcc_test(2, 95) \
+    "defined __MWERKS__ && __MWERKS__ <= 0x32FF @ cw9" \
+    "defined _MSC_VER && _MSC_VER < 1300 && !defined UNDER_CE @ vc6" \
+    "defined _MSC_VER && _MSC_VER < 1300 && defined UNDER_CE @ evc4" \
+    "defined __MWERKS__ && __MWERKS__ <= 0x31FF @ cw8"
+  do
+    boost_tag_test=`expr "X$i" : 'X\([[^@]]*\) @ '`
+    boost_tag=`expr "X$i" : 'X[[^@]]* @ \(.*\)'`
+    AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+#if $boost_tag_test
+/* OK */
+#else
+# error $boost_tag_test
+#endif
+]])], [boost_cv_lib_tag=$boost_tag; break], [])
+  done
+AC_LANG_POP([C++])dnl
+  case $boost_cv_lib_tag in #(
+    # Some newer (>= 1.35?) versions of Boost seem to only use "gcc" as opposed
+    # to "gcc41" for instance.
+    *-gcc | *'-gcc ') :;; #(  Don't re-add -gcc: it's already in there.
+    gcc*)
+      boost_tag_x=
+      case $host_os in #(
+        darwin*)
+          if test $boost_major_version -ge 136; then
+            # The `x' added in r46793 of Boost.
+            boost_tag_x=x
+          fi;;
+      esac
+      # We can specify multiple tags in this variable because it's used by
+      # BOOST_FIND_LIB that does a `for tag in -$boost_cv_lib_tag' ...
+      boost_cv_lib_tag="$boost_tag_x$boost_cv_lib_tag -${boost_tag_x}gcc"
+      ;; #(
+    unknown)
+      AC_MSG_WARN([[could not figure out which toolset name to use for $CXX]])
+      boost_cv_lib_tag=
+      ;;
+  esac
+fi])dnl end of AC_CACHE_CHECK
+])# _BOOST_FIND_COMPILER_TAG
+
+
+# _BOOST_GUESS_WHETHER_TO_USE_MT()
+# --------------------------------
+# Compile a small test to try to guess whether we should favor MT (Multi
+# Thread) flavors of Boost.  Sets boost_guess_use_mt accordingly.
+AC_DEFUN([_BOOST_GUESS_WHETHER_TO_USE_MT],
+[# Check whether we do better use `mt' even though we weren't ask to.
+AC_LANG_PUSH([C++])dnl
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+#if defined _REENTRANT || defined _MT || defined __MT__
+/* use -mt */
+#else
+# error MT not needed
+#endif
+]])], [boost_guess_use_mt=:], [boost_guess_use_mt=false])
+AC_LANG_POP([C++])dnl
+])
+
+# _BOOST_AC_LINK_IFELSE(PROGRAM, [ACTION-IF-TRUE], [ACTION-IF-FALSE])
+# -------------------------------------------------------------------
+# Fork of _AC_LINK_IFELSE that preserves conftest.o across calls.  Fragile,
+# will break when Autoconf changes its internals.  Requires that you manually
+# rm -f conftest.$ac_objext in between to really different tests, otherwise
+# you will try to link a conftest.o left behind by a previous test.
+# Used to aggressively optimize BOOST_FIND_LIB (see the big comment in this
+# macro).
+#
+# Don't use "break" in the actions, as it would short-circuit some code
+# this macro runs after the actions.
+m4_define([_BOOST_AC_LINK_IFELSE],
+[m4_ifvaln([$1], [AC_LANG_CONFTEST([$1])])dnl
+rm -f conftest$ac_exeext
+boost_save_ac_ext=$ac_ext
+boost_use_source=:
+# If we already have a .o, re-use it.  We change $ac_ext so that $ac_link
+# tries to link the existing object file instead of compiling from source.
+test -f conftest.$ac_objext && ac_ext=$ac_objext && boost_use_source=false &&
+  _AS_ECHO_LOG([re-using the existing conftest.$ac_objext])
+AS_IF([_AC_DO_STDERR($ac_link) && {
+         test -z "$ac_[]_AC_LANG_ABBREV[]_werror_flag" ||
+         test ! -s conftest.err
+       } && test -s conftest$ac_exeext && {
+         test "$cross_compiling" = yes ||
+         $as_executable_p conftest$ac_exeext
+dnl FIXME: use AS_TEST_X instead when 2.61 is widespread enough.
+       }],
+      [$2],
+      [if $boost_use_source; then
+         _AC_MSG_LOG_CONFTEST
+       fi
+       $3])
+ac_objext=$boost_save_ac_objext
+ac_ext=$boost_save_ac_ext
+dnl Delete also the IPA/IPO (Inter Procedural Analysis/Optimization)
+dnl information created by the PGI compiler (conftest_ipa8_conftest.oo),
+dnl as it would interfere with the next link command.
+rm -f core conftest.err conftest_ipa8_conftest.oo \
+      conftest$ac_exeext m4_ifval([$1], [conftest.$ac_ext])[]dnl
+])# _BOOST_AC_LINK_IFELSE
+
+# Local Variables:
+# mode: autoconf
+# End:
diff --git a/provision/compile.sh b/provision/compile.sh
new file mode 100644
index 0000000..c13143a
--- /dev/null
+++ b/provision/compile.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+start=$(date +%s)
+echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
+cd /vagrant
+
+echo "::::::::::::: COMPILE HEDGEHOG :::::::::::::"
+cd /vagrant
+./autogen.sh
+mkdir build
+cd build
+../configure
+make
+
+echo "::::::::::::: INSTALL HEDGEHOG :::::::::::::"
+make install
+make install-rpg
+
+
+echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
+
+cd /vagrant
+
+end=$(date +%s)
+
+diff=$(( $end - $start ))
+
+
+echo ":::::::::::::::::::::::::::::::::"
+echo "::::: COMPILATION: $diff s"
+echo ":::::::::::::::::::::::::::::::::"
diff --git a/provision/dependencies.sh b/provision/dependencies.sh
new file mode 100644
index 0000000..dd4db92
--- /dev/null
+++ b/provision/dependencies.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+
+start=$(date +%s)
+echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
+cd /vagrant
+
+echo "::::::::::::: REPO :::::::::::::"
+add-apt-repository ppa:opencpu/rapache
+apt-get update
+
+echo "::::::::::::: C++ :::::::::::::"
+apt-get install -y libboost-all-dev libtool libpqxx-dev
+
+echo "::::::::::::: R :::::::::::::"
+apt-get install -y r-base r-base-core r-base-dev libcairo2-dev libxt-dev r-cran-ggplot2 r-cran-dbi r-cran-cairodevice r-cran-reshape r-cran-digest
+echo "install.packages(c('brew','Cairo','googleVis','RPostgreSQL','R.utils','yaml','dplyr'), repos='http://cran.rstudio.com/')" | R --vanilla
+
+echo "::::::::::::: POSTGRES :::::::::::::"
+apt-get install -y postgresql postgresql-client postgresql-server-dev-9.3 postgresql-contrib pgbouncer
+
+echo "::::::::::::: Apache, etc. :::::::::::::"
+apt-get install -y apache2-mpm-prefork apache2-prefork-dev git libapache2-mod-r-base libyaml-tiny-perl
+
+echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
+
+cd /vagrant
+
+end=$(date +%s)
+
+diff=$(( $end - $start ))
+
+echo ":::::::::::::::::::::::::::::::::"
+echo "::::: DEPENDENCIES: $diff s"
+echo ":::::::::::::::::::::::::::::::::"
diff --git a/provision/launch.sh b/provision/launch.sh
new file mode 100644
index 0000000..1361701
--- /dev/null
+++ b/provision/launch.sh
@@ -0,0 +1,50 @@
+#!/bin/bash
+
+start=$(date +%s)
+echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
+cd /vagrant
+
+echo "::::::::::::: CREATE USERS :::::::::::::"
+# create a system user and group called hedgehog
+addgroup --system hedgehog
+adduser --system --ingroup hedgehog hedgehog
+
+chown -R hedgehog:hedgehog /usr/local/var/hedgehog/data
+chown -R www-data:www-data /usr/local/var/hedgehog/www
+
+echo "::::::::::::: CONFIGURE APACHE :::::::::::::"
+a2dissite 000-default.conf
+
+cp /usr/local/share/hedgehog/conf/hedgehog.conf /etc/apache2/sites-available/
+a2ensite hedgehog.conf
+
+echo "
+<Directory /usr/local/share/hedgehog>
+AllowOverride None
+Require all granted
+</Directory>
+
+<Directory /usr/local/var/hedgehog/www>
+AllowOverride None
+Require all granted
+</Directory>" >> /etc/apache2/apache2.conf
+
+#apache/rapache write some of their logs to user.* so it can be useful to change the syslog config: Uncomment the line beginning 'user.*'.
+#sudo vi /etc/rsyslog.d/50-default.conf
+
+service apache2 restart
+
+#Edit the the <prefix>/etc/hedgehog/nodes.csv to specify the servers, nodes and grouping to be used (example format is provided with entries commented out).
+#sudo -u postgres /usr/local/bin/database_create
+#sudo -u hedgehog /usr/local/bin/hedgehogctl database_init
+echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"
+
+cd /vagrant
+
+end=$(date +%s)
+
+diff=$(( $end - $start ))
+
+echo ":::::::::::::::::::::::::::::::::"
+echo "::::: CONFIGURATION: $diff s"
+echo ":::::::::::::::::::::::::::::::::"
diff --git a/src/DSCDataManager.cpp b/src/DSCDataManager.cpp
new file mode 100755
index 0000000..f116a14
--- /dev/null
+++ b/src/DSCDataManager.cpp
@@ -0,0 +1,366 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/**
+ */
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+
+#include <string>
+#include <map>
+#include <exception>
+#include <iostream>
+#include <time.h>
+
+#include <boost/property_tree/ptree.hpp>
+#include <boost/property_tree/xml_parser.hpp>
+#include <boost/foreach.hpp>
+#include <boost/algorithm/string.hpp>
+#include <boost/filesystem.hpp>
+
+#include "pqxx/pqxx"
+
+#include "dsc_types.h"
+#include "DSCDataManager.h"
+#include "DSCStrategy.h"
+#include "DSCStrategyFactory.h"
+
+#include "Format1DSCStrategy.h"
+#include "Format2DSCStrategy.h"
+#include "Format3aDSCStrategy.h"
+#include "Format3bDSCStrategy.h"
+#include "Format4DSCStrategy.h"
+
+using namespace std;
+
+DSCDataManager::DSCDataManager(const string& server, const string& node, pqxx::work *pg_db_trans) {
+
+    // Use the transaction pointer to decide if the output will be a database
+    if ( pg_db_trans == NULL ) {
+        using_pg_db_ = false;
+        pg_db_transaction_ = NULL;
+        server_ = server;
+        node_ = node;
+    } else {
+        using_pg_db_ = true;
+        pg_db_transaction_ = pg_db_trans;
+        server_ = server;
+        node_ = node;
+        server_id_ = get_server_id(server, *pg_db_trans);
+        node_id_ = get_node_id(server_id_, node, *pg_db_trans);
+    }
+
+}
+
+void
+DSCDataManager::clear() {
+
+    start_time_ = 0;
+    name_.clear();
+    time_strings_[0].clear();
+    time_strings_[1].clear();
+    time_strings_[2].clear();
+    counts_.clear();
+    preprocess_required_ = false;
+    for(vector<DSCStrategy*>::iterator strategy_it = strategies_.begin(); strategy_it != strategies_.end(); ++strategy_it){
+        delete *strategy_it;
+    }
+    strategies_.clear();
+
+}
+
+int
+DSCDataManager::load(const string dat_line, vector<DSCStrategy*>::iterator strategy_it) {
+
+    clear();
+
+    // Call out to the strategy factory as the strategies 
+    // know how to convert the line of data to a DSCUMap
+    if ( using_pg_db_ == true ){
+        if ( (*strategy_it)->process_dat_line(counts_, time_strings_, dat_line) != 0 ) return 1;
+        struct tm *tmp_time;
+        time_t tmp_time1;
+        tmp_time1 = (time_t)atoi(time_strings_[1].c_str());
+        tmp_time = gmtime(const_cast<const time_t *>(&tmp_time1));
+        char yyyymmdd_c[9];
+        char yyyy_mm_c[8];
+        strftime(yyyymmdd_c, 9, "%Y%m%d", tmp_time);
+        time_strings_[0].assign(yyyymmdd_c,8);
+        strftime(yyyy_mm_c, 8, "%Y_%m", tmp_time);
+        time_strings_[2].assign(yyyy_mm_c,7);
+    } else {
+        cerr << "Error: Attempt to write a dat file to a dat file for graph " << name_ << endl;
+        exit(1);
+    }
+    return 0;
+    
+}
+
+int
+DSCDataManager::load(const boost::filesystem::path& file_path, vector<DSCStrategy*>::iterator strategy_it) {
+
+    clear();
+
+    name_ = file_path.filename().string();
+    
+    struct tm tm;
+    std::stringstream ss;
+    string tmp_time_string;
+    char yyyy_mm_c[8];
+
+    //  The only time/date we know of is that contained in the path 
+    //  to the dat file.
+    time_strings_[0].assign(file_path.parent_path().filename().string());
+    tmp_time_string = time_strings_[0] + " 00:00:00";
+    if ( strptime(tmp_time_string.c_str(), "%Y%m%d %H:%M:%S", &tm) != NULL ) {
+        ss << mktime(&tm);
+        time_strings_[1] = ss.str();
+    strftime(yyyy_mm_c, 8, "%Y_%m", &tm);
+    time_strings_[2].assign(yyyy_mm_c,7);
+
+    } else {
+        cerr << "Error: Unable to extract unixtime when loading dat file " << file_path.string() << endl;
+        return 1;
+    }
+
+    // Call out to the strategy factory as the strategies 
+    // know how to convert the file to a DSCUMap
+    if ( using_pg_db_ == true ){
+        (*strategy_it)->process_dat_file(counts_, time_strings_, file_path);
+    } else {
+        cerr << "Error: Attempt to write a dat file to a dat file for graph " << name_ << endl;
+        exit(1);
+    }
+    return 0;
+
+}
+
+int 
+DSCDataManager::load(const boost::property_tree::ptree::value_type &array_value, bool rssac) {
+
+    int dim_counter = 0; 
+    string dim_type[2];
+    string dim1_value;
+    string dim2_value;
+
+    dim_type[0].clear();
+    dim_type[1].clear();
+    dim1_value.clear();
+    dim2_value.clear();
+    
+    clear();
+    preprocess_required_ = true;
+
+    // catch any generic XML processing errors
+    try {
+	    if ( boost::equals ( array_value.first, "array") ) {
+	        BOOST_FOREACH(boost::property_tree::ptree::value_type const& value, array_value.second) {
+	            // Get attributes from the array element
+	            if ( boost::equals ( value.first, "<xmlattr>") ) {
+	                name_       = value.second.get<string>("name");
+	                start_time_ = value.second.get<int>("start_time");
+
+	                // round start time down to start of the minute
+	                start_time_ = int(start_time_ / 60) * 60;
+
+	                struct tm *tmp_time;
+	                time_t tmp_time1;
+	                tmp_time1 = (time_t)start_time_;
+	                tmp_time = gmtime(const_cast<const time_t *>(&tmp_time1));
+	                char yyyymmdd_c[9];
+	                char yyyy_mm_c[8];
+	                strftime(yyyymmdd_c, 9, "%Y%m%d", tmp_time);
+	                time_strings_[0].assign(yyyymmdd_c,8);
+	                // don't use strftime here due to timezone issues
+	                ostringstream tmp_time2;
+	                tmp_time2 << start_time_;
+	                time_strings_[1] = tmp_time2.str();
+	                strftime(yyyy_mm_c, 8, "%Y_%m", tmp_time);
+	                time_strings_[2].assign(yyyy_mm_c,7);
+	            }
+
+	            // Get dimensions (Should always be 2 of them)
+	            if ( boost::equals ( value.first, "dimension") ) {
+	                BOOST_FOREACH(boost::property_tree::ptree::value_type const& dim_value, value.second) {
+	                    if ( boost::equals ( dim_value.first, "<xmlattr>") ) {
+	                        dim_type[dim_counter] = dim_value.second.get<string>("type");
+	                        dim_counter++;
+	                    }
+	                }
+	            }
+
+	            // Get data
+	            if ( boost::equals ( value.first, "data") ) {
+	                BOOST_FOREACH(boost::property_tree::ptree::value_type const& value1, value.second) {
+	                    if ( boost::equals ( value1.first, dim_type[0]) ) {
+	                        BOOST_FOREACH(boost::property_tree::ptree::value_type const& value2, value1.second) {
+	                            if ( boost::equals ( value2.first, "<xmlattr>") ) {
+	                                dim1_value = value2.second.get<string>("val");
+	                            } else if ( boost::equals ( value2.first, dim_type[1]) ) {
+	                                int tmp_count = 0;
+	                                BOOST_FOREACH(boost::property_tree::ptree::value_type const& value3, value2.second) {
+	                                    if ( boost::equals ( value3.first, "<xmlattr>") ) {
+	                                        dim2_value = value3.second.get<string>("val");
+	                                        tmp_count  = value3.second.get<int>("count");
+	                                    }
+	                                }
+	                                DSCKey tmp_key(dim1_value,dim2_value);
+	                                counts_.insert(pair<DSCKey,int>(tmp_key, tmp_count));
+	                            }
+	                        }
+	                    }
+	                }
+	            }
+	        }
+	    }
+	}
+    catch( std::exception & e )
+    {
+        cerr << "Exception loading XML file: " << e.what() << endl;
+        return 1;
+    }
+    catch( ... )
+    {
+        cerr << "Unknown exception caught" << endl;
+        return 1;
+    }
+
+  strategies_ = DSCStrategyFactory::createStrategy(server_, name_, rssac);
+  return 0;
+
+}
+
+int
+DSCDataManager::process() {
+
+    int status = 0;
+
+    // if no data, don't process
+    if(counts_.size() == 0)
+        return 0;
+    
+    long unsigned int i=0;
+    vector<DSCStrategy*>::iterator strategy_it;
+    
+    for(strategy_it = strategies_.begin(); strategy_it != strategies_.end(); strategy_it++){
+        DSCUMap *output_counts = &counts_;
+        DSCUMap new_counts;
+        // preprocess is only required if the input channel was XML
+        if (preprocess_required_) {
+           (*strategy_it)->preprocess_data(counts_, new_counts, time_strings_, server_id_, node_id_, pg_db_transaction_);
+           output_counts = &new_counts;
+        }
+        if ( using_pg_db_ == true ) {
+            status = (*strategy_it)->process_to_db(*output_counts, time_strings_, server_id_, node_id_, *pg_db_transaction_);
+            if (status) return 1;
+        }
+        if ( using_pg_db_ == false ) {
+            (*strategy_it)->process_to_dat(*output_counts, time_strings_);
+        }
+        i++;
+    }
+    return 0;
+
+}
+
+int
+DSCDataManager::process(vector<DSCStrategy*>::iterator strategy_it) {
+
+    int status=0;
+    // if no data, don't process
+    if(counts_.size() == 0)
+        return 0;
+
+    DSCUMap *output_counts = &counts_;
+    DSCUMap new_counts;
+    if (preprocess_required_){
+       (*strategy_it)->preprocess_data(counts_, new_counts, time_strings_, server_id_, node_id_, pg_db_transaction_);
+       output_counts = &new_counts;
+    }
+    if ( using_pg_db_ == true ) {
+        status = (*strategy_it)->process_to_db(*output_counts, time_strings_, server_id_, node_id_, *pg_db_transaction_);
+        if (status) return 1;
+    } else {
+        cerr << "Error: Expected to use a database but no transaction found." << endl;
+        exit(1);
+    }
+    return 0;
+
+}
+
+int
+DSCDataManager::get_server_id(const string& server, pqxx::work& pg_db_trans) {
+
+    stringstream sql;
+    try {
+        sql.clear();
+        sql.str("");
+        sql << "SELECT id FROM dsc.server WHERE name = '" << server << "'" << endl;
+        pqxx::result r = pg_db_trans.exec(sql.str());
+        if ( r.size() != 1 ) {
+            cerr << "Error: Expected 1 server with name " << server << ", "
+                 << "but found " << r.size() << endl;
+            exit( EXIT_FAILURE );
+        }
+        return r[0][0].as<int>();
+    }
+    catch( runtime_error & e )
+    {
+        cerr << "Runtime error: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Exception: " << e.what() << sql.str() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+
+}
+
+int
+DSCDataManager::get_node_id(int server_id, const string& node, pqxx::work& pg_db_trans) {
+
+    stringstream sql;
+    try {
+        sql.clear();
+        sql.str("");
+        sql << "SELECT id FROM dsc.node WHERE name = '" << node << "' AND server_id = '" << server_id << "'" << endl;
+        pqxx::result r = pg_db_trans.exec(sql.str());
+        if ( r.size() != 1 ) {
+            cerr << "Error: Expected 1 node with name " << node << " and server ID " << server_id << ", "
+                 << "but found " << r.size() << endl;
+            exit( EXIT_FAILURE );
+        }
+        return r[0][0].as<int>();
+    }
+    catch( runtime_error & e )
+    {
+        cerr << "Runtime error: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Exception: " << e.what() << sql.str() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+
+}
diff --git a/src/DSCDataManager.h b/src/DSCDataManager.h
new file mode 100755
index 0000000..9e78354
--- /dev/null
+++ b/src/DSCDataManager.h
@@ -0,0 +1,101 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#include <string>
+#include <map>
+#include <time.h>
+
+#include <boost/property_tree/ptree.hpp>
+#include <boost/unordered_map.hpp>
+
+#include "pqxx/pqxx"
+
+#include "dsc_types.h"
+
+using namespace std;
+
+class DSCStrategy;
+
+
+/// \brief Class that parses the DSC data and calls an appropriate strategy to 
+/// to deal with the data.
+/// 
+/// This class is responsible for owning the data and providing load functions 
+/// for the various input channels and provides functions to process the data 
+/// using an appropriate strategy.
+class DSCDataManager
+{
+    
+private:
+    
+    // ****************
+    // Member data
+    // ****************
+    /// \brief string to hold the name of the array
+    std::string name_;
+
+    /// \brief integer start time read from the xml
+    int start_time_;
+
+    /// \brief array of strings to hold time is a variety of formats
+    std::string time_strings_[3];
+    /// \brief unordered map (defined in dsc_types.h) containing the data
+    /// loaded from the xml
+    DSCUMap counts_;
+    /// \brief A vetor holding the strategies returned by the factory
+    std::vector<DSCStrategy*> strategies_;
+    /// \brief Boolean to tell if the output is a database or a dat file
+    bool using_pg_db_;
+    /// \brief Pointer to a PostgreSQL transaction.
+    pqxx::work *pg_db_transaction_;
+    /// \brief The name of the server node.
+    std::string node_;
+    /// \brief The name of the server.
+    std::string server_;
+    
+    int node_id_;
+    int server_id_;
+    bool preprocess_required_;
+    int get_server_id(const string& server, pqxx::work& pg_db_trans);
+    int get_node_id(int server_id, const string& node, pqxx::work& pg_db_trans);
+	
+public:
+    
+    // ****************
+    // Constructor
+    // ****************
+    /// \brief Initialize the class making use of the pointer to the database 
+    /// transaction to determine the output destination.
+    //  TODO: allow configure to decide if db functionality is compiled in
+    DSCDataManager(const string& server, const string& node, pqxx::work *pg_db_trans);
+            
+    /// \brief Method to clear the member data. This class is only constructed 
+    /// once and repeatedly reused by the DSCIOManager.
+    void clear();
+    
+    /// \brief Method to parse the xml held in a boost::property_tree::ptree
+    ///        and populate the member data.
+    int load(const boost::property_tree::ptree::value_type &array_value, bool rssac);
+    /// \brief Alternate load method that can parse an entire dat file into the 
+    /// member data.
+    int load(const boost::filesystem::path&, vector<DSCStrategy*>::iterator strategy_it);
+    /// \brief Alternate load method that can parse a single line from a dat 
+    /// file into the member data.
+    int load(const std::string dat_line, vector<DSCStrategy*>::iterator strategy_it);
+    /// \brief Process method that is used when processing xml input via  
+    /// multiple strategies.
+    int process();
+    /// \brief Alternate process method that uses a specific strategy to process
+    /// the data.
+    int process(vector<DSCStrategy*>::iterator strategy_it);
+    
+};
\ No newline at end of file
diff --git a/src/DSCIOManager.cpp b/src/DSCIOManager.cpp
new file mode 100644
index 0000000..fd9414d
--- /dev/null
+++ b/src/DSCIOManager.cpp
@@ -0,0 +1,521 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   xml_files.cpp
+ */
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+
+#include <exception>
+#include <iostream>
+#include <algorithm>
+#include <vector>
+#include <exception>
+#include <time.h>
+
+#include <boost/property_tree/ptree.hpp>
+#include <boost/property_tree/xml_parser.hpp>
+#include <boost/foreach.hpp>
+#include <boost/filesystem.hpp>
+
+#include <pqxx/pqxx>
+
+#include "DSCIOManager.h"
+#include "DSCDataManager.h"
+#include "dsc_types.h"
+#include "DSCStrategyFactory.h"
+
+#include "config.h"
+
+using namespace std;
+namespace bfs = boost::filesystem;
+
+// Helper function
+// Get current date/time, format is YYYY-MM-DD HH:mm:ss time-zone
+const std::string currentDateTime() {
+
+    time_t     now = time(0);
+    struct tm  tstruct;
+    char       buf[80];
+    tstruct = *localtime(&now);
+    strftime(buf, sizeof(buf), "%Y-%m-%d %X %Z", &tstruct);
+
+    return buf;
+
+}
+
+void replace_string(std::string& str, const std::string& from, const std::string& to) {
+    if(from.empty())
+        return;
+    size_t start_pos = 0;
+    while((start_pos = str.find(from, start_pos)) != std::string::npos) {
+        str.replace(start_pos, from.length(), to);
+        start_pos += to.length(); // In case 'to' contains 'from', like replacing 'x' with 'yx'
+    }
+}
+
+
+DSCIOManager::DSCIOManager(const enum INPUT_OUTPUT input, const enum INPUT_OUTPUT output, const string& conn_string, const string& processing_start_date_string, bool rssac) {
+
+    input_ = input;
+    output_ = output;
+    conn_string_ = conn_string;
+    pg_db_conn_ = NULL;
+    pg_db_trans_ = NULL;
+    processing_start_date_ = NULL;
+    rssac_=rssac;
+
+    cout << endl << endl << "----------------------- " << currentDateTime() << " ----------------------" << endl;
+    // work out if we have a valid start date
+    struct tm p_start_date  = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, NULL };
+    if ( ! processing_start_date_string.empty() ) {
+       cout << "Processing start date defined as " << processing_start_date_string << endl; 
+       if ( strptime(processing_start_date_string.c_str(), "%Y-%m-%d", &p_start_date) != NULL ) {   
+                processing_start_date_ = new time_t(mktime(&p_start_date));
+                //cout << "Time is: " << p_start_date.tm_year << " " << p_start_date.tm_mon << " " << p_start_date.tm_mday << " " << p_start_date.tm_hour << " " << p_start_date.tm_min << " " << p_start_date.tm_sec << endl;   
+       } else {
+           cerr << "Error: Invalid processing start date \'" << processing_start_date_string << "\' provided. Must be YYYY-MM-DD format. Exiting." << endl;
+           exit(1);
+       }
+    } 
+}
+
+DSCIOManager::~DSCIOManager() {
+    
+    cout << "----------------------- " << currentDateTime() << " -----------------------" << endl << endl;
+    if ( processing_start_date_ != NULL ) {
+        delete processing_start_date_;
+    }
+}
+    
+int
+DSCIOManager::dsc_connect_pg_db() {
+    pg_db_conn_ = new pqxx::connection( conn_string_ );
+    if ( pg_db_conn_ == NULL ) {
+        cerr << "Error connecting to database." << endl;
+        exit(1);
+    }
+    pg_db_conn_->activate();
+    
+    /* Check Database version */
+    int db_version = 0;
+    try {
+        pg_db_trans_ = new pqxx::work( *pg_db_conn_ );
+        if ( pg_db_trans_ == NULL ) {
+            cerr << "Error starting transaction." << endl;
+            exit(1);
+        }
+        const pqxx::result R = pg_db_trans_->exec("select version from version;");
+        pqxx::result::const_iterator row;
+        for (row = R.begin(); row != R.end(); ++row) {
+            row["version"].to(db_version);
+        }
+        if ( db_version != DB_SCHEMA_VERSION ) {
+            cout << "Database version is not supported. Expected: " << DB_SCHEMA_VERSION << " Found: " << db_version << endl;
+            exit( EXIT_FAILURE);
+        }
+        if ( pg_db_trans_ != NULL ) {
+            pg_db_trans_->commit();
+            delete pg_db_trans_;
+            pg_db_trans_ = NULL;
+        }
+    }
+    catch ( pqxx::unique_violation & e )
+    {
+        cerr << "Unique_violation error: " << e.what() << endl;
+        exit( EXIT_FAILURE);
+    }
+    catch( runtime_error & e )
+    {
+        cerr << "Runtime error: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Exception: " << e.what() << " select version from version;" << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+    
+    return 0;
+
+}
+
+int
+DSCIOManager::dsc_disconnect_pg_db() {
+
+    pg_db_conn_->disconnect();
+    delete pg_db_conn_;
+    // TODO Errors
+    return 0;
+
+}
+
+void
+DSCIOManager::dsc_activate_ouput_destination() {
+
+    switch (output_) {
+        case DAT:
+            // Nothing to do
+            break;
+        case PG_DB:
+            dsc_connect_pg_db();
+            cout << "*** Database connected" << endl;
+            break;
+        default:
+            // Should never get here!
+            cerr << "Error: Unknown output destination." << endl;
+            exit(1);
+            break;
+    }
+}
+
+void
+DSCIOManager::dsc_deactivate_ouput_destination() {
+    switch (output_) {
+        case DAT:
+            // Nothing to do
+            break;
+        case PG_DB:
+            dsc_disconnect_pg_db();
+            cout << "*** Database disconnected" << endl;
+            break;
+        default:
+            // Should never get here!
+            cerr << "Error: Unknown output destination." << endl;
+            exit(1);
+            break;
+    }
+
+}
+
+void
+DSCIOManager::dsc_import_input_from_source() {
+
+    dsc_populate_files_vector();
+    int file_counter=file_vector_.size();
+    int file_failures = 0;
+    int file_commit_status = 0;  // status of transaction commit
+    int data_process_status = 0; // status of data processing
+    
+    string node = bfs::initial_path().filename().generic_string();
+    string server = bfs::initial_path().parent_path().filename().generic_string();
+    //fix potential sql injection
+    replace(node.begin(), node.end(), '\'', '_');
+    replace(server.begin(), server.end(), '\'', '_');
+    //transform to 'internal' name
+    replace(server.begin(), server.end(), '-', '_');
+    replace_string(server, ".", "__");
+    transform(server.begin(), server.end(), server.begin(), ::tolower);
+    
+    cout << "*** Processing server: " << server << endl;
+    cout << "*** Processing node:   " << node << endl;
+    cout << "*** Found:             " << file_counter << " " <<  file_extension_ << " files to process." << endl;
+    
+    for (std::vector<bfs::path>::const_iterator file_vector_it (file_vector_.begin()); file_vector_it != file_vector_.end(); ++file_vector_it) {
+        if ( pg_db_conn_ != NULL ) {
+            pg_db_trans_ = new pqxx::work( *pg_db_conn_ );
+            if ( pg_db_trans_ == NULL ) {
+                cerr << "Error starting transaction." << endl;
+                exit(1);
+            }
+        }
+        cout << endl << currentDateTime() << ": Starting:  " << (*file_vector_it).string() << endl;
+        DSCDataManager* dsc_data_manager = new DSCDataManager(server, node, pg_db_trans_);
+        file_commit_status = 0;
+        
+        switch (input_) {
+            case XML: {
+                // Partially read the xml. This is done using a policy tree,
+                // which uses rapidxml to do the parsing. There is no validation
+                // performed. Each array element is a "data unit" to send to the
+                // data manager load function.
+                dsc_xml_ = new boost::property_tree::ptree();
+                // If reading fails, exception is thrown.
+                try {
+                  boost::property_tree::read_xml((*file_vector_it).string(), *dsc_xml_);
+                  // We can't validate the XML but make this call the check that the file 
+                  // at least has the correct top level node
+                  try {
+                    dsc_xml_->get_child("dscdata");
+                  }
+                  catch( std::exception & e ) {
+                    cerr << "Error: XML file has no content: " << e.what() << endl;
+                    data_process_status = 1;
+                    break;					
+                  }
+                }
+                catch( std::exception & e )
+                {
+                    cerr << "Exception reading XML file: " << e.what() << endl;
+                    data_process_status = 1;
+                    break;
+                }
+                catch( ... )
+                {
+                    cerr << "Unknown exception caught" << endl;
+                    data_process_status = 1;
+                    break;
+                }
+                // Get each array element and process it
+                BOOST_FOREACH(boost::property_tree::ptree::value_type &array_element, dsc_xml_->get_child("dscdata")) {
+                    data_process_status = dsc_data_manager->load(array_element, rssac_);
+                    if (data_process_status) break;
+                    data_process_status = dsc_data_manager->process();
+                    if (data_process_status) break;
+                }
+                delete dsc_xml_;
+            }
+            break;
+            case DAT: {
+                // Call out to strategies to find out how this dat file 
+                // needs to be processed.
+                // F1, F2 and F3a dat files contain multiple "data units"
+                vector<DSCStrategy*>::iterator strategy_it;
+                string name = (*file_vector_it).filename().string();
+                std::vector<DSCStrategy*> strategies_vector = DSCStrategyFactory::createStrategyDat(server, name);
+                bool multi;
+                
+                if (strategies_vector.size() != 1 ) { 
+                    cerr << "Error: Failed to find a single strategy for loading dat file " << (*file_vector_it).string() << " (found: " << strategies_vector.size() << ")" << endl;
+                    // exit(1);
+                    break;
+                }
+                
+                strategy_it = strategies_vector.begin();
+                multi = (*strategy_it)->is_dat_file_multi_unit();
+
+                if ( multi ) {
+                    // read the file one line at a time
+                    bfs::fstream dat_file;
+                    string dat_line;
+                    dat_file.open((*file_vector_it), ios_base::in);
+                    while (getline(dat_file, dat_line)) {
+                        if ( dsc_data_manager->load(dat_line, strategy_it) !=0 ) continue;
+                        data_process_status = dsc_data_manager->process(strategy_it);
+                        if (data_process_status) break;
+                    }
+                } else {
+                    // The entire file is one "data unit"
+                    dsc_data_manager->load((*file_vector_it), strategy_it);
+                    data_process_status = dsc_data_manager->process(strategy_it);
+                    if (data_process_status) break;
+                }
+            }
+            break;
+            default:
+                // Should never get here!
+                cerr << "Error: Unknown input source" << endl;
+                exit(1);
+        }
+        
+        if ( pg_db_trans_ != NULL ) {  
+                if ( data_process_status == 0 ) {
+                    try {
+                        pg_db_trans_->commit();
+                    }             
+                    catch( std::exception & e ) {
+                        cerr << "Commit failed with exception message: " << e.what() << endl;    
+                        file_commit_status = 1;             
+                    }
+                    catch( ... ) {
+                        cerr << "Commit failed with general error" << endl;
+                        file_commit_status = 1;
+                    }
+                }
+                // Tidy up. If the processing was not successful the transaction will implicitly rollback
+                delete pg_db_trans_;
+                pg_db_trans_ = NULL;         
+        }      
+        
+        delete dsc_data_manager;        
+        
+        // move the file to the done directory if all is well
+        if (file_commit_status == 0 && data_process_status == 0 ) {
+             dsc_move_file_to_done((*file_vector_it));        
+        } else {
+              dsc_handle_failed_file((*file_vector_it), data_process_status);
+            file_failures++;
+        }
+                
+        // Display helpful message. Flush to ensure that it is seen.
+        file_counter--;
+        cout << currentDateTime() << ": Completed: " << setw(50) << left << (*file_vector_it).string() << "\t" << file_counter << " remaining (" << file_failures << " failures)."<< endl;
+
+        std::flush(cout);
+    }
+    
+    // clear the file vector
+    file_vector_.clear();
+    
+    // endl since the previous cout used \r
+    cout << endl;
+
+}
+
+void
+DSCIOManager::dsc_populate_files_vector() {
+    
+    std::string start_of_search_path;
+    switch (input_) {
+        case XML: 
+            start_of_search_path = "./incoming";
+            file_extension_ = ".xml";
+            break;
+        case DAT:
+            start_of_search_path = ".";
+            file_extension_ = ".dat";
+            break;
+        default:
+            // Should never get here!
+            cerr << "Error: Unknown input source" << endl;
+            exit(1);
+            break;
+    }
+    
+    bfs::path p_start (start_of_search_path);
+    bfs::path::iterator path_it;
+
+    // iterators for iterating over all the paths/files in the data directory
+    bfs::recursive_directory_iterator it(p_start), end_it;
+    try {
+        if (bfs::exists(p_start)) {
+            if (bfs::is_directory(p_start)) {
+                BOOST_FOREACH( bfs::path const &p_search, make_pair(it, end_it) ) {                 
+                    if ( p_search.string().find("done") == string::npos ) {
+                        if ( ! bfs::is_directory(p_search) ) {
+                            if ( p_search.extension() == file_extension_ ) {
+                                if (! exclude_file(p_search, start_of_search_path) ) {
+                                        file_vector_.push_back(p_search);                                   
+                                }
+                            }
+                        }
+                    }
+                }
+                sort(file_vector_.begin(), file_vector_.end());
+            } else {
+                cout << p_start << " exists, but is not a directory\n";
+                exit(1);
+            }
+        } else {
+            cout << p_start << " does not exist\n";
+            exit(1);
+        }
+    }
+    catch (const bfs::filesystem_error& ex) {
+        cout << "FS Error: " << ex.what() << endl;
+        exit(1);
+    }
+    catch (std::exception &e) {
+        cerr << "Error creating file vector: " << e.what() << endl;
+        exit(1);
+    }
+    
+}
+
+void
+DSCIOManager::dsc_move_file_to_done(bfs::path path_src) {
+    
+    bfs::path path_done ("./done");
+    std::string done_sub_dir;
+    bfs::path path_date;
+    bfs::path path_done_date;
+    bfs::path path_newfile;
+    
+    try {
+        switch (input_) {
+            case XML: 
+                done_sub_dir = "dscdata";
+                path_date = path_src.parent_path().stem(); //./incoming/yyyymmdd/*.xml --> yyyymmdd
+                break;
+            case DAT:
+                done_sub_dir = "dat";
+                path_date = path_src.parent_path().stem(); // ./yyyymmdd/*.dat --> yyyymmdd
+                break;
+            default:
+                // Should never get here!
+                cerr << "Error: Unknown input source" << endl;
+                exit(1);
+                break;
+        }
+        path_done_date = path_done/path_date/done_sub_dir;
+        if (!bfs::exists(path_done_date)) {
+                bfs::create_directories(path_done_date);
+        }
+        path_newfile = path_done_date/path_src.filename();
+        bfs::rename(path_src, path_newfile);
+        cout << currentDateTime() << ": Moved:     " << path_src.string() << " to " <<  path_newfile.string() << endl;
+    }
+    catch (const bfs::filesystem_error& ex) {
+        cout << "FS Error: " << ex.what() << endl;
+        exit(1);
+    }
+    catch (std::exception &e) {
+        cerr << "Error moving file to done: " << e.what() << endl;
+        exit(1);
+    }
+ 
+}
+
+void
+DSCIOManager::dsc_handle_failed_file(bfs::path path_src, int process) {
+    
+    bfs::path path_newfile;  
+    std::string failed = ".failed.";
+    std::string reason_string = process ? " Failed to process data " : "Failed to commit data to DB";
+    
+    try {
+        path_newfile = path_src.string() + failed + bfs::unique_path().string();
+        bfs::rename(path_src, path_newfile);
+        cout << currentDateTime() << ": Error: failed to process: " << path_src.string() << "Reason: " << reason_string << endl;
+        cout << currentDateTime() << ": File renamed to:          " << path_newfile.string() << endl;
+    }
+    catch (const bfs::filesystem_error& ex) {
+        cout << "FS Error: " << ex.what() << endl;
+        exit(1);
+    }
+    catch (std::exception &e) {
+        cerr << "Error handling failed file: " << e.what() << endl;
+        exit(1);
+    }
+ 
+}
+
+bool 
+DSCIOManager::exclude_file(bfs::path path_src, const string& start_of_search_path) {
+
+    if ( input_ != XML || processing_start_date_ == NULL ) {
+        return false;
+    }
+
+    // Only do this test on files under './incoming'
+    if ( path_src.parent_path().parent_path().string().compare(start_of_search_path) == 0 ) {
+        struct tm tm  = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, NULL };
+        if ( strptime(path_src.parent_path().filename().string().c_str(), "%Y-%m-%d", &tm) != NULL ) {
+            time_t dir_time = mktime(&tm);
+            //cout << "Diff time is " << difftime(*processing_start_date_, dir_time) << endl ;
+            if ( difftime(*processing_start_date_, dir_time) > 0 )  {
+                cout << "File " << path_src.string() << " excluded from processing as it is earlier than the processing start date" << endl;
+                return true;
+            } else {
+                return false;
+            }
+        }                     
+    }
+    return false;
+
+}
\ No newline at end of file
diff --git a/src/DSCIOManager.h b/src/DSCIOManager.h
new file mode 100755
index 0000000..e125433
--- /dev/null
+++ b/src/DSCIOManager.h
@@ -0,0 +1,92 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#ifndef DSCIOMANAGER_H
+#define	DSCIOMANAGER_H
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+
+#include <vector>
+
+#include <boost/property_tree/ptree.hpp>
+#include <boost/filesystem.hpp>
+
+#include <pqxx/pqxx>
+
+#include "dsc_types.h"
+
+namespace bfs = boost::filesystem;
+
+/// \brief DSC IO Manager class.
+/// 
+/// This class is responsible for configuring input and output channels.
+/// Provides public functions to activate and deactivate the output destination
+/// and to import the data from the source. It only does minimal parsing of any 
+/// input data in order to determine how to proceed. The loading of the data 
+/// is left to the data manager.
+class DSCIOManager
+{
+private:
+    
+    // ****************
+    // Member data
+    // ****************	
+    enum INPUT_OUTPUT input_;
+    enum INPUT_OUTPUT output_;
+    std::string conn_string_;
+    std::string file_extension_;
+    std::vector<bfs::path> file_vector_;
+    pqxx::connection *pg_db_conn_;
+    pqxx::work *pg_db_trans_;
+    time_t *processing_start_date_;
+    bool rssac_;
+    // This is not the intended use of a property tree but it works for me
+    boost::property_tree::ptree *dsc_xml_;
+    
+    /// Populate a vector of filenames.
+    void dsc_populate_files_vector();
+    /// Move data files that have been successfully processed into a done dir.
+    void dsc_move_file_to_done(bfs::path path_src);
+    /// Decide what to do with a failed file
+    void dsc_handle_failed_file(bfs::path path_src, int process);
+    ///  Function to test if a file should be excluded from processing
+    bool exclude_file(bfs::path path_src, const string& start_of_search_path);
+    /// Connect to PostgreSQL database.
+    int dsc_connect_pg_db();
+    /// Disconnect from PostgreSQL database.
+    int dsc_disconnect_pg_db();
+
+
+public:
+    
+    // ****************
+    // Constructor
+    // ****************
+    /// \brief Initialize the class. Sets some of the member variables.
+    DSCIOManager(const enum INPUT_OUTPUT input, const enum INPUT_OUTPUT output, const string& conn_string, const string& processing_start_date_string, bool rssac);
+    virtual ~DSCIOManager();
+    /// \brief Activates the output destination (at the moment this only does 
+    /// something for the database case).
+    void dsc_activate_ouput_destination();
+    /// \brief Deactivates the output destination (at the moment this only does 
+    /// something for the database case).
+    void dsc_deactivate_ouput_destination();
+    /// \brief Import data from either XML or DAT files and calls the data 
+    /// manager to process and output the data.
+    void dsc_import_input_from_source();
+};
+
+#endif
\ No newline at end of file
diff --git a/src/DSCStrategy.cpp b/src/DSCStrategy.cpp
new file mode 100755
index 0000000..e56d983
--- /dev/null
+++ b/src/DSCStrategy.cpp
@@ -0,0 +1,350 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#include <iostream>
+#include <string>
+#include <set>
+#include <map>
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/unordered_map.hpp>
+#include <boost/lexical_cast.hpp>
+
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+
+#include "pqxx/pqxx"
+
+using namespace std;
+namespace bfs = boost::filesystem;
+
+DSCStrategy::DSCStrategy(string server, string name, string keys[], int keylength) {
+
+    if ( keylength != 0 ) keys_.insert (keys, keys+keylength);
+    server_name_ = server;
+    plot_name_ = name;
+
+}
+
+bool
+DSCStrategy::is_valid_time(const string& time_string) {
+
+    bool res = true;
+    try {
+        boost::lexical_cast<time_t>(time_string);
+    if (time_string.size() != 10 ) res = false;
+    }
+    catch (boost::bad_lexical_cast &e) {
+        res = false;
+    }
+    
+    return(res);
+
+}
+
+void
+DSCStrategy::elsify_unwanted_keys(DSCUMap &dscdata) {
+
+    // elsify_unwanted_keys
+    DSCUMap::iterator it=dscdata.begin();
+    
+    boost::unordered_map<string,int> tmp_value;
+    boost::unordered_map<string,int>::iterator it1;
+    while ( it != dscdata.end() ) {
+        // skip if map key (second part) is else
+        // skip if map key (second part) is in the list of valid 
+        // keys for this graph
+        if ( ((*it).first.second == "else") || (keys_.find((*it).first.second) != keys_.end()) ) {
+            it++;
+            continue;
+        }
+        // Sum the values of the unknown keys 
+        // and then delete the unknown key.
+        // Don't forget we are in an iterator and so must post increment it
+        tmp_value[(*it).first.first] += (*it).second;
+        dscdata.erase(it++);
+    }
+    
+    it1 = tmp_value.begin();
+    for(; it1 != tmp_value.end(); it1++) {
+        DSCKey tmp_key1((*it1).first,"else");
+        if ((*it1).second !=  0) {
+            dscdata[tmp_key1] += (*it1).second;
+        }
+    }
+
+}
+
+int
+DSCStrategy::accum1d_to_count(const DSCUMap &dscdata) {
+
+    int count=0;
+    DSCUMap::const_iterator it=dscdata.begin();
+    for ( ; it!=dscdata.end() ; it++){
+        if ((*it).first.second == "-:SKIPPED_SUM:-") {
+            continue;
+        } else if ((*it).first.second == "-:SKIPPED:-") {
+            count += (*it).second;
+        } else {
+            count++;
+        }
+    }
+    return count;
+
+}
+
+void
+DSCStrategy::merge_accum1d(const DSCUMap &dscdata, DSCUMap &accum){
+
+    DSCKey tmp_key;
+    tmp_key.first = "All";
+    DSCUMap::const_iterator it=dscdata.begin();
+    for ( ; it!=dscdata.end() ; it++){
+        tmp_key.second = (*it).first.second;
+        accum[tmp_key] += (*it).second;
+    }
+
+}
+
+// convert a "2D" accum-type dataset to a "count-type"
+// ie, count all input->{*}{k2} into output->{k2}
+void
+DSCStrategy::accum2d_to_count(const DSCUMap &dscdata, DSCUMap &count) {
+    
+    DSCUMap::const_iterator it=dscdata.begin();
+    DSCKey tmp_key;
+    while ( it != dscdata.end() ) {
+        if ( (*it).first.second == "-:SKIPPED_SUM:-" ) { 
+       
+        } else if ( (*it).first.second == "-:SKIPPED:-" ) {
+            tmp_key.first = "All";
+            tmp_key.second = (*it).first.first;
+            count[tmp_key] += (*it).second;
+        } else {
+            tmp_key.first = "All";
+            tmp_key.second = (*it).first.first;
+            count[tmp_key] += 1;
+        }
+        it++;
+    }
+
+}
+
+// convert a "2D" accum-type dataset to a trace-type
+// ie, sum all input->{*}{k2} into output->{k2}
+void
+DSCStrategy::accum2d_to_trace(const DSCUMap &dscdata, DSCUMap &trace){
+
+    DSCUMap::const_iterator it=dscdata.begin();
+    DSCKey tmp_key;
+    while ( it != dscdata.end() ) {
+        if ( (*it).first.second != "-:SKIPPED:-" ) {
+            tmp_key.first = "All";
+            tmp_key.second = (*it).first.first;
+            trace[tmp_key] += (*it).second;
+        }
+        it++;
+        
+    }
+
+}
+
+void
+DSCStrategy::swap_dimensions(const DSCUMap &dscdata, DSCUMap &swap) {
+
+    DSCUMap::const_iterator it=dscdata.begin();
+    DSCKey tmp_key;
+    while ( it != dscdata.end() ) {
+        tmp_key.first = (*it).first.second;
+        tmp_key.second = (*it).first.first;
+        swap[tmp_key] = (*it).second;
+        it++;
+    }
+    if ( swap.size() != dscdata.size() ) cout << "Error swapping keys\n";
+
+}
+
+void
+DSCStrategy::merge_accum2d(const DSCUMap &dscdata, DSCUMap &old) {
+
+    DSCUMap::const_iterator it=dscdata.begin();
+    for ( ; it != dscdata.end() ; it++ ) old[(*it).first] += (*it).second;
+
+}
+
+void
+DSCStrategy::trim_accum2d(DSCUMap &dscdata) {
+
+    // key order has been swapped before getting here
+    // dscdata -> (TLD, QTYPE) -> count)
+    multimap<int,string> r_map;
+    multimap<int,string>::reverse_iterator ir;
+    set<DSCKey> keys2delete;
+    
+    // iterate over allowed keys
+    set<string>::iterator key = keys_.begin();
+    for (; key != keys_.end() ; key++ ) {
+        
+        int n = 0;
+        // iterate over dscdata and create multimap
+        // r_map = count -> TLD (sorted))
+        DSCUMap::iterator it=dscdata.begin();
+        for (; it != dscdata.end() ; it++) {   
+            if ((*it).first.second != (*key) ) continue;
+            // use multimap to sort by value
+            r_map.insert(pair<int,string>((*it).second,(*it).first.first));
+        }
+        // iterate over r_map
+        ir=r_map.rbegin();
+        for (; ir != r_map.rend() ; ir++) {
+            // Create key_pair
+            // TLD,allowed key
+            DSCKey key_pair ((*ir).second,*key);
+            // continue if n <= 1000 
+            //  this means that we have seen 1000 or more (TLD,allowed key) pairs
+            if ( ++n <= 1000 ) continue;
+            // continue if key_pair is not found in dscdata
+            if ( dscdata.find(key_pair) == dscdata.end() ) continue;
+            // so now we have TLD,allowed key pairs above 1000
+            // for each one add one to dscdata("-:SKIPPED:-", allowed key)  value
+            DSCKey tmp_skip ("-:SKIPPED:-",*key);
+            dscdata[tmp_skip]++;        
+            // and add the value to a dscdata("-:SKIPPED_SUM:-", allowed key)  value
+            DSCKey tmp_skip_sum ("-:SKIPPED_SUM:-",*key);
+            dscdata[tmp_skip_sum] += dscdata[key_pair];
+            // Add the (TLD,allowed key pair) to a set of keys to be deleted later
+            keys2delete.insert(key_pair);
+        }
+        r_map.clear();
+        // and finally delete the dscdata entries in set keys2delete
+        set<DSCKey>::iterator it1 = keys2delete.begin();
+        for (; it1 != keys2delete.end(); it1++) {
+            dscdata.erase((*it1));
+        }
+        keys2delete.clear();
+    }
+    return;
+
+}
+
+void
+DSCStrategy::open_file(string dtime[], ios_base::openmode open_mode) {
+
+    bfs::path p_tmp;
+    bfs::path p_tmp_file;
+    bfs::path p_unique;
+    bfs::path p_data_root = ".";
+    bfs::path p_storage = dtime[0];
+    bfs::path p_data_storage(p_data_root/p_storage);
+    bfs::path p_filename(plot_name_ + ".dat");
+    bfs::path p_dat_file(p_data_storage/p_filename);
+    
+    if ( open_mode & ios_base::app || open_mode & ios_base::trunc || open_mode & ios_base::out) {
+        p_unique = bfs::unique_path();
+        p_tmp_file = plot_name_ + ".dat." + p_unique.string();
+        p_tmp = p_data_storage/p_tmp_file;
+    }
+
+    try {
+        
+        if (! bfs::exists(p_data_storage)) bfs::create_directory(p_data_storage);
+        if (! bfs::exists(p_data_storage)) {
+            cerr <<  "FS Error: " << p_data_storage << " directory could not be created." << endl;
+            return;
+        }
+        if (! bfs::is_directory(p_data_storage)) {
+            cerr <<  "FS Error: " <<  p_data_storage << " exists but is not a directory." << endl;
+            return;
+        }
+        
+        if ( open_mode & ios_base::app || open_mode & ios_base::trunc || open_mode & ios_base::out) {
+            if (bfs::exists(p_dat_file)) bfs::copy_file(p_dat_file, p_tmp);
+            datafile_.filename=p_dat_file.string();
+            datafile_.tmpfilename=p_tmp.string();
+            datafile_.open_mode = open_mode;
+            datafile_.filestream.open(p_tmp, open_mode);
+            if (! datafile_.filestream.is_open()) {
+                cerr <<  "FS Error: " << p_tmp << " temporary file did not open." << endl;
+                return;
+            }
+        } else if ( open_mode & ios_base::in ) {
+            datafile_.filename=p_dat_file.string();
+            datafile_.open_mode = open_mode;
+            datafile_.filestream.open(p_dat_file, open_mode);
+            if (bfs::exists(p_dat_file) && (! datafile_.filestream.is_open()))  {
+                cerr <<  "FS Error: " << p_dat_file << " dat file did not open." << endl;
+                return;
+            }
+        } else {
+            cerr << "FS Error: un-supported file open mode" << endl;
+        }
+    }
+    catch (const bfs::filesystem_error& ex) {
+        cerr << "FS Error: " << ex.what() << endl;
+    }
+    return;
+
+}
+
+void
+DSCStrategy::close_file() {
+
+    try {
+        datafile_.filestream.close();
+        if (datafile_.open_mode & ios_base::app || datafile_.open_mode & ios_base::trunc || datafile_.open_mode & ios_base::out) {
+            bfs::rename(datafile_.tmpfilename, datafile_.filename);
+        }
+    }
+    catch (const bfs::filesystem_error& ex) {
+        cerr << "FS Error: " << ex.what() << endl;
+    }
+
+}
+
+int
+DSCStrategy::get_plot_id(const string& plot, pqxx::work& pg_db_trans) {
+
+    stringstream sql;
+    pqxx::result r;
+    
+    try {
+        sql.clear();
+        sql.str("");
+        sql << "SELECT id FROM dsc.dataset WHERE name = '" << plot << "'" << endl;
+        r = pg_db_trans.exec(sql.str());
+        if ( r.size() != 1 ) {
+            cerr << "Error: Expected 1 plot with name " << plot << ", "
+                 << "but found " << r.size() << endl;
+            exit( EXIT_FAILURE );
+        }
+    }
+    catch( runtime_error & e )
+    {
+        cerr << "Runtime error: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Exception: " << e.what() << sql.str() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+    return r[0][0].as<int>();
+
+}
+
diff --git a/src/DSCStrategy.h b/src/DSCStrategy.h
new file mode 100755
index 0000000..5177d49
--- /dev/null
+++ b/src/DSCStrategy.h
@@ -0,0 +1,79 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   DSCStrategy.h
+ */
+
+// TODO(asap): Doxygen comments
+
+#ifndef DSCSTRATEGY_H
+#define	DSCSTRATEGY_H
+
+#include <string>
+#include <set>
+#include <map>
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/unordered_map.hpp>
+
+#include <dsc_types.h>
+
+#include <pqxx/pqxx>
+
+using namespace std;
+namespace bfs = boost::filesystem;
+
+class DSCStrategy {
+
+public:
+    
+    DSCStrategy() {};
+    DSCStrategy(string server, string name, string keys[], int keylength);
+    
+    virtual ~DSCStrategy() {};
+    virtual void process_to_dat(const DSCUMap &, string dtime[]) = 0;
+    virtual int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) = 0;
+    virtual int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) = 0;
+    virtual void process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &file_path) = 0;
+    virtual void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans) = 0;
+    virtual bool is_dat_file_multi_unit() = 0;
+    
+protected:
+    
+    set<string> keys_;
+    string server_name_;
+    string plot_name_;
+    DSCdatfile datafile_;
+    
+    virtual void write_dat(const DSCUMap &, string dtime[]) = 0;
+    
+    void elsify_unwanted_keys(DSCUMap &dscdata);
+    void accum2d_to_count(const DSCUMap &dscdata, DSCUMap &count);
+    void accum2d_to_trace(const DSCUMap &dscdata, DSCUMap &trace);
+    void swap_dimensions(const DSCUMap &dscdata, DSCUMap &swap);
+    void merge_accum2d(const DSCUMap &dscdata, DSCUMap &old);
+    void open_file(string dtime[], ios_base::openmode open_mode);
+    void close_file();
+    void trim_accum2d(DSCUMap  &dscdata);
+    int accum1d_to_count(const DSCUMap &dscdata);
+    void merge_accum1d(const DSCUMap &dscdata, DSCUMap &accum);
+    bool is_valid_time(const string& time_string);    
+    int get_plot_id(const string& plot, pqxx::work& pg_db_trans);
+    void add_1_to_skipped(const DSCUMap &dscdata, DSCUMap &newdata);
+    
+};
+
+#endif	/* DSCSTRATEGY_H */
+
diff --git a/src/DSCStrategyFactory.cpp b/src/DSCStrategyFactory.cpp
new file mode 100755
index 0000000..6295220
--- /dev/null
+++ b/src/DSCStrategyFactory.cpp
@@ -0,0 +1,427 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   DSCStrategyFactory.cpp
+ */
+
+// TODO(refactor): This should be re-factored and pick up the data from a config file. 
+// TODO(refactor): A config file should also be used to specify which data to process and which to ignore.
+
+#include <iostream>
+#include <string>
+#include <vector>
+#include "dsc_types.h"
+#include "DSCStrategyFactory.h"
+#include "DSCStrategy.h"
+#include "Format1DSCStrategy.h"
+#include "Format2DSCStrategy.h"
+#include "Format3aDSCStrategy.h"
+#include "Format3bDSCStrategy.h"
+#include "Format4DSCStrategy.h"
+
+using namespace std;
+
+DSCStrategyFactory::DSCStrategyFactory() {
+
+}
+
+DSCStrategyFactory::~DSCStrategyFactory() {
+
+}
+
+vector<DSCStrategy*>
+DSCStrategyFactory::createStrategy(const string &server, const string &name, bool rssac) {
+
+    vector<DSCStrategy*> return_v;
+
+    //  Format 1
+    if (name.compare("rcode") == 0) {
+        string keys[1] = { "dummy" };
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name, keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("server_addr") == 0) {
+        string keys[1] = { "dummy" };
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name, keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("qtype") == 0) {
+        string keys[11] = { "1", "2", "5", "6", "12", "15", "28", "33",
+                                "38", "255",  "else"};
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,11);
+        return_v.push_back( strat );
+        
+        string keys1[8] = {"24", "25", "30", "43", "46", "47", "50", "48"};
+        Format1DSCStrategy* strat1 = new Format1DSCStrategy(server,"dnssec_qtype",keys1,8);
+        return_v.push_back( strat1 );
+        return return_v;        
+    
+    } else if (name.compare("opcode") == 0) {
+        string keys[6] = {"0", "1", "2", "4", "5", "else"};
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,6);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("edns_version") == 0) {
+        string keys[3] = {"none", "0", "else"};
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,3);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("rd_bit") == 0) {
+        string keys[2] = {"set", "clr"};
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,2);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("idn_qname") == 0) {
+        string keys[2] = {"normal", "idn"};
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,2);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("do_bit") == 0) {
+        string keys[2] = {"set", "clr"};
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,2);
+        return_v.push_back( strat );
+        return return_v;
+    //  End Format 1
+
+    //  Format 2
+    } else if (name.compare("server_addr_vs_trans") == 0) {
+        string keys[3] = {"dummy"};
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,0);
+        return_v.push_back( strat );
+        return return_v; 
+    
+    } else if (name.compare("certain_qnames_vs_qtype") == 0) {
+        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
+                                "38", "255", "else"};
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,11);
+        return_v.push_back( strat );
+        return return_v;
+    
+    } else if (name.compare("direction_vs_ipproto") == 0) {
+        string keys[3] = {"icmp", "tcp", "udp"};
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,3);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("chaos_types_and_names") == 0) {
+        string keys[3] = {"hostname.bind", "version.bind", "other"};
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,3);
+        return_v.push_back( strat );
+        return return_v;
+    
+    } else if (name.compare("pcap_stats") == 0) {
+        string keys[1] = {"dummy"};
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,0);
+        return_v.push_back( strat );
+        return return_v;
+
+    } else if (name.compare("transport_vs_qtype") == 0) {
+        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
+                                "38", "255", "else"};
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,11);
+        return_v.push_back( strat );
+        return return_v;
+    } else if ( name.compare("traffic_volume_queries") == 0 ||
+        name.compare("traffic_volume_responses") == 0 ) {
+        if (rssac) {
+            string keys[1]= { "dummy" };
+            Format2DSCStrategy* strat = new Format2DSCStrategy(server, name, keys,0);
+            return_v.push_back( strat );
+            return return_v;        
+        } else {
+            return return_v;
+    }
+    //  End Format 2 
+       
+    //  Format 3
+    } else if (name.compare("client_subnet") == 0) {
+        string keys[1] = {"dummy"};
+        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server, "client_subnet_count",keys,0);
+        return_v.push_back( strat );
+        Format3bDSCStrategy* strat1 = new Format3bDSCStrategy(server, "client_subnet_accum",keys,0);
+        return_v.push_back( strat1 );
+        return return_v;
+      
+    } else if (name.compare("idn_vs_tld") == 0) {
+        string keys[1] = {"dummy"};
+        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server, name,keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    
+    } else if (name.compare("ipv6_rsn_abusers") == 0) {
+        string keys[1] = {"dummy"};
+        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server, "ipv6_rsn_abusers_count",keys,0);
+        return_v.push_back( strat );        
+        Format3bDSCStrategy* strat1 = new Format3bDSCStrategy(server,"ipv6_rsn_abusers_accum",keys,0);
+        return_v.push_back( strat1 );
+        return return_v;
+
+    } else if (name.compare("traffic_sizes_queries") == 0 ||
+        name.compare("traffic_sizes_responses") == 0) {
+        if (rssac) {
+            string keys[1] = {"dummy"};
+            Format3bDSCStrategy* strat = new Format3bDSCStrategy(server, name,keys,0);
+            return_v.push_back( strat );
+            return return_v;
+        } else {
+            return return_v;
+        }
+
+    } else if (name.compare("unique_sources") == 0) {
+        if (rssac) {
+            string keys[1] = {"dummy"};
+            Format3bDSCStrategy* strat = new Format3bDSCStrategy(server, "unique_sources_raw",keys,0);
+            return_v.push_back( strat );
+            return return_v;    
+        } else {
+            return return_v;   
+        }
+    //  End Format 3
+        
+    //  Format 4
+    } else if (name.compare("qtype_vs_tld") == 0) {
+        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
+                                "38", "255", "else"};
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,11);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("client_addr_vs_rcode") == 0) {
+        string keys[1] = {"dummy"};
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"client_addr_vs_rcode_accum",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("qtype_vs_client_subnet") == 0) {
+        string keys[1] = {"dummy"};
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"qtype_vs_client_subnet",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("qtype_vs_qnamelen") == 0) {
+        string keys[1] = {"dummy"};
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("rcode_vs_replylen") == 0) {
+        string keys[1] = {"dummy"};
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,0);
+        return_v.push_back( strat );
+        return return_v;       
+        
+    // } else if (name.compare("client_subnet_vs_tld") == 0) {
+    //     string keys[1] = {"dummy"};
+    //     Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,0);
+    //     return_v.push_back( strat );
+    //     return return_v;
+        
+    //  End format 4
+        
+    //  Multiple formats        
+    } else if (name.compare("client_subnet2") == 0) {
+        string keys[11] = {"ok", "non-auth-tld", "root-servers.net",
+                                "localhost", "a-for-a", "a-for-root",
+                                "rfc1918-ptr", "funny-qclass", "funny-qtype", 
+                                "src-port-zero", "malformed"};
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"client_subnet2_trace",keys,11);
+        return_v.push_back( strat );
+        
+        Format1DSCStrategy* strat1 = new Format1DSCStrategy(server,"client_subnet2_count",keys,11);
+        return_v.push_back( strat1 );
+        
+        Format4DSCStrategy* strat2 = new Format4DSCStrategy(server,"client_subnet2_accum",keys,11);
+        return_v.push_back( strat2 );
+        return return_v;
+        
+    } else if (name.compare("dns_ip_version_vs_qtype") == 0) {
+        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
+                                "38", "255", "else"};
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"dns_ip_version",keys,11);
+        return_v.push_back( strat );
+        
+        Format4DSCStrategy* strat1 = new Format4DSCStrategy(server,"dns_ip_version_vs_qtype",keys,11);
+        return_v.push_back( strat1 );
+        return return_v;
+    //  End Multiple formats
+
+    } else {
+        // return empty vector
+        return return_v;
+    }
+
+}
+
+vector<DSCStrategy*>
+DSCStrategyFactory::createStrategyDat(const string &server, const string &name) {
+
+    vector<DSCStrategy*> return_v;
+    string keys[1] = {"dummy"};
+
+    //  Format 1    
+    if (name.compare("rcode.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"rcode", keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("qtype.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"qtype",keys,0);
+        return_v.push_back( strat );
+        return return_v;        
+    
+    } else if (name.compare("dnssec_qtype.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"dnssec_qtype",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    
+    } else if (name.compare("opcode.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"opcode",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("edns_version.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"edns_version",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("rd_bit.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"rd_bit",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("idn_qname.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"idn_qname",keys,0);
+        return_v.push_back( strat );
+        return return_v;        
+        
+    } else if (name.compare("do_bit.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"do_bit",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    //  End Format 1 
+
+    //  Format 2
+    } else if (name.compare("certain_qnames_vs_qtype.dat") == 0) {
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"certain_qnames_vs_qtype",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    
+    } else if (name.compare("direction_vs_ipproto.dat") == 0) {
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"direction_vs_ipproto",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("chaos_types_and_names.dat") == 0) {
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"chaos_types_and_names",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    
+    } else if (name.compare("pcap_stats.dat") == 0) {
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"pcap_stats",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+
+    } else if (name.compare("transport_vs_qtype.dat") == 0) {
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"transport_vs_qtype",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    //  End Format 2 
+
+    //  Format 3
+    } else if (name.compare("client_subnet_count.dat") == 0) {
+        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server,"client_subnet_count",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("client_subnet_accum.dat") == 0) {
+        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server,"client_subnet_accum",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("idn_vs_tld.dat") == 0) {
+        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server,"idn_vs_tld",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("ipv6_rsn_abusers_count.dat") == 0) {
+        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server,"ipv6_rsn_abusers_count",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("ipv6_rsn_abusers_accum.dat") == 0) {
+        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server,"ipv6_rsn_abusers_accum",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    //  End Format 3
+
+    //  Format 4
+    } else if (name.compare("qtype_vs_tld.dat") == 0) {
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"qtype_vs_tld",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("client_addr_vs_rcode_accum.dat") == 0) {
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"client_addr_vs_rcode_accum",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("qtype_vs_qnamelen.dat") == 0) {
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"qtype_vs_qnamelen",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("rcode_vs_replylen.dat") == 0) {
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"rcode_vs_replylen",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    //  End format 4
+        
+    //  Multiple formats
+    } else if (name.compare("client_subnet2_trace.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"client_subnet2_trace",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    
+    } else if (name.compare("client_subnet2_count.dat") == 0) {
+        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"client_subnet2_count",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("client_subnet2_accum.dat") == 0) {
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"client_subnet2_accum",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("dns_ip_version_vs_qtype.dat") == 0) {
+        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"dns_ip_version_vs_qtype",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+        
+    } else if (name.compare("dns_ip_version.dat") == 0) {
+        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"dns_ip_version",keys,0);
+        return_v.push_back( strat );
+        return return_v;
+    //  End Multiple formats
+
+    } else {
+        // return empty vector
+        return return_v;
+    }
+
+}
diff --git a/src/DSCStrategyFactory.h b/src/DSCStrategyFactory.h
new file mode 100644
index 0000000..71b3e79
--- /dev/null
+++ b/src/DSCStrategyFactory.h
@@ -0,0 +1,35 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   DSCStrategyFactory.h
+ */
+
+#ifndef DSCSTRATEGYFACTORY_H
+#define	DSCSTRATEGYFACTORY_H
+
+#include <string>
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+
+using namespace std;
+
+class DSCStrategyFactory {
+    DSCStrategyFactory();
+    ~DSCStrategyFactory();
+public:
+    static vector<DSCStrategy*> createStrategy(const string &server, const string &name, bool rssac);
+    static vector<DSCStrategy*> createStrategyDat(const string &server, const string &name);
+};
+
+#endif	/* DSCSTRATEGYFACTORY_H */
+
diff --git a/src/Format1DSCStrategy.cpp b/src/Format1DSCStrategy.cpp
new file mode 100644
index 0000000..785ba1a
--- /dev/null
+++ b/src/Format1DSCStrategy.cpp
@@ -0,0 +1,177 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#include <iostream>
+#include <map>
+#include <string>
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/tokenizer.hpp>
+#include "dsc_types.h"
+#include "Format1DSCStrategy.h"
+#include "DSCStrategy.h"
+
+#include "pqxx/pqxx"
+
+using namespace std;
+namespace bfs = boost::filesystem;
+
+//
+// time k1 v1 k2 v2 ...
+//
+
+Format1DSCStrategy::Format1DSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
+    
+}
+
+void
+Format1DSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string /*dtime*/[], int /*server*/, int /*node*/, pqxx::work* /*pg_db_trans*/) {
+    
+    if ( plot_name_ == "client_subnet2_count") {
+        accum2d_to_count(orig_dscdata, new_dscdata);
+    } else if ( plot_name_ == "client_subnet2_trace") {
+        accum2d_to_trace(orig_dscdata, new_dscdata);
+    } else if ( plot_name_ == "client_port_range" || 
+                plot_name_ == "edns_bufsiz"  ||
+                plot_name_ == "rcode" ||
+                plot_name_ == "server_addr") {
+        new_dscdata = orig_dscdata;
+    } else {
+        new_dscdata = orig_dscdata;
+        elsify_unwanted_keys(new_dscdata);    
+    }
+
+}
+
+void
+Format1DSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {
+
+    open_file(dtime, ios_base::app | ios_base::out);
+    write_dat(dscdata, dtime);
+    close_file();
+
+}
+
+int
+Format1DSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    return write_db(dscdata, dtime, server_id, node_id, pg_db_trans);
+
+}
+
+void
+Format1DSCStrategy::write_dat(const DSCUMap &my_dscdata, string dtime[]){
+    
+    // Now store the new data in the file
+    datafile_.filestream << dtime[1];
+    
+    DSCUMap::const_iterator it1=my_dscdata.begin();
+    while ( it1 != my_dscdata.end() ) {
+        if ( (*it1).second != 0 ) {
+            datafile_.filestream << " " << (*it1).first.second << " " << (*it1).second;    
+        }
+        it1++;
+    }
+
+    datafile_.filestream << endl;
+
+}
+
+int
+Format1DSCStrategy::process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) {
+
+    DSCKey tmp_key;
+    tmp_key.first = "All";
+    typedef boost::tokenizer<boost::char_separator<char> > t_tokenizer;
+    boost::char_separator<char> sep(" ");
+    
+    if ( dat_line.length() == 0 ) {
+        cerr << endl << "Found empty line in dat file - skipped" << endl;
+        return 1;
+    }
+    t_tokenizer tok(dat_line, sep);
+
+    // Read the time and the alternating keys and values
+    int i=0;
+    t_tokenizer::iterator field=tok.begin();
+    // Read time from first field
+    time_strings[1] = *field;
+    if (! is_valid_time(time_strings[1]) ) return 1;
+    field++;
+    for(; field!=tok.end();++field){
+        i++;
+        if ( i%2 != 0 ) tmp_key.second = *field;
+        if ( i%2 == 0 ) {
+            int tmp_value;
+            stringstream(*field) >> tmp_value;
+            counts[tmp_key] = tmp_value;
+        }
+    }
+    return 0;
+
+}
+
+int
+Format1DSCStrategy::write_db(const DSCUMap &my_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans){
+
+    DSCUMap::const_iterator it1=my_dscdata.begin();
+    stringstream sql;
+    sql.clear();
+    int plot_id = get_plot_id(plot_name_, pg_db_trans);
+    sql.str("");    
+    sql << "INSERT INTO dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\" (server_id, node_id, plot_id, starttime, key1, key2, value) values"; 
+    while ( 1 ) {
+        if ( (*it1).second != 0 ) {
+            sql << " (" << server_id << ", " << node_id << ", " << plot_id 
+                << ", timestamptz 'epoch' + " << dtime[1] 
+                <<  " * interval '1 second', '" << (*it1).first.second << "', 'NONE', " 
+                << (*it1).second << ")";
+
+            it1++;
+            if (it1 == my_dscdata.end() ) {
+                sql << ";";
+                break;
+            } else {
+                sql << ",";
+            }
+        } else {
+            cerr << "Format 1 Error: Bad data at " << plot_name_ << " " << dtime[1] << ". Skipping..." << endl;
+            return 1;
+        }
+    }
+    try {
+        pg_db_trans.exec(sql.str());
+    }
+    catch ( pqxx::unique_violation & e )
+    {
+        cerr << "Format 1 unique_violation error: " << e.what() << endl;
+        return 1;
+    }
+    catch( runtime_error & e )
+    {
+        cerr << "Format 1 runtime error: " << e.what() << endl;
+        return 1;
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Format 1 exception: " << e.what() << sql.str() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Format 1 Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+    return 0;
+}
diff --git a/src/Format1DSCStrategy.h b/src/Format1DSCStrategy.h
new file mode 100644
index 0000000..8efa7a8
--- /dev/null
+++ b/src/Format1DSCStrategy.h
@@ -0,0 +1,49 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   Format1DSCStrategy.h
+ */
+
+#ifndef FORMAT1DSCSTRATEGY_H
+#define FORMAT1DSCSTRATEGY_H
+#include <string>
+#include <map>
+#include "dsc_types.h"
+#include"DSCStrategy.h"
+
+using namespace std;
+
+class Format1DSCStrategy: public DSCStrategy {
+
+public:
+
+    Format1DSCStrategy(string server, string name, string keys[], int keylength);
+    virtual ~Format1DSCStrategy() {};
+
+    void process_to_dat(const DSCUMap &, string dtime[]);
+    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line);
+    void process_dat_file(DSCUMap &/*counts*/, std::string /*time_strings*/[], const boost::filesystem::path &/*file_path*/) {};
+    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
+    bool is_dat_file_multi_unit() { return true; };
+    
+private:
+
+    void write_dat(const DSCUMap &, string dtime[]);
+    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+
+};
+
+
+#endif	/* FORMAT1DSCSTRATEGY_H */
+
diff --git a/src/Format2DSCStrategy.cpp b/src/Format2DSCStrategy.cpp
new file mode 100644
index 0000000..37e55f3
--- /dev/null
+++ b/src/Format2DSCStrategy.cpp
@@ -0,0 +1,206 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+
+#include <iostream>
+#include <map>
+#include <string>
+
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/tokenizer.hpp>
+
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "Format2DSCStrategy.h"
+
+#include "pqxx/pqxx"
+
+using namespace std;
+namespace bfs = boost::filesystem;
+
+Format2DSCStrategy::Format2DSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
+    
+}
+
+void
+Format2DSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string /*dtime*/[], int /*server*/, int /*node*/, pqxx::work* /*pg_db_trans*/) {
+
+      if ( plot_name_.compare("traffic_volume_queries") == 0 || 
+                plot_name_.compare("traffic_volume_responses") == 0  ||
+                plot_name_.compare("traffic_sizes_queries") == 0 ||
+                plot_name_.compare("traffic_sizes_responses") == 0 ) {
+        new_dscdata = orig_dscdata;
+    } else if ( ! keys_.empty() ) {
+        new_dscdata = orig_dscdata;
+        elsify_unwanted_keys(new_dscdata);    
+    } else {
+        swap_dimensions(orig_dscdata, new_dscdata);
+    }
+
+}
+
+void
+Format2DSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {
+    
+    open_file(dtime, ios_base::app | ios_base::out);
+    write_dat(dscdata, dtime);
+    close_file();
+
+}
+
+int
+Format2DSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+     return write_db(dscdata, dtime, server_id, node_id, pg_db_trans);
+
+}
+
+void
+Format2DSCStrategy::write_dat(const DSCUMap &my_dscdata, string dtime[]) {
+    
+    DSCUMap::const_iterator it;
+    map<string,string> tmp_kv;
+    map<string,string>::iterator does_key_exist;
+    // Don't forget the map's key is a pair as well
+    for ( it=my_dscdata.begin() ; it != my_dscdata.end(); it++ ){
+        string t = static_cast<ostringstream*>( &(ostringstream() << (*it).second) )->str();
+        does_key_exist = tmp_kv.find((*it).first.first); 
+        if ( does_key_exist == tmp_kv.end() ) {
+            tmp_kv[(*it).first.first] +=  (*it).first.second + ":" + t;
+        } else {
+            tmp_kv[(*it).first.first] +=  ":" + (*it).first.second + ":" + t;
+        }
+    }
+    if (! tmp_kv.empty() ){
+        datafile_.filestream << dtime[1];
+        map<string,string>::iterator it1;
+        for ( it1=tmp_kv.begin() ; it1 != tmp_kv.end(); it1++ ){
+            datafile_.filestream << " " << (*it1).first << " " << (*it1).second;
+        }
+        datafile_.filestream << endl;
+    }
+
+}
+
+int
+Format2DSCStrategy::process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) {
+
+    // reads a 2-level hash database WITH time dimension
+    // ie: time k1 (k:v:k:v) k2 (k:v:k:v)
+    // each line comes from a single XML array
+    // Need to return time and DSCUMap for each line
+    
+    string k;
+    DSCKey tmp_key;
+    typedef boost::tokenizer<boost::char_separator<char> > t_tokenizer;
+    boost::char_separator<char> sep(" ");
+    boost::char_separator<char> sep1(":");
+
+    if ( dat_line.length() == 0 )  {
+        cerr << endl << "Found empty line in dat file - skipped" << endl;
+        return 1;
+    }
+
+    t_tokenizer tok(dat_line, sep);
+    t_tokenizer::iterator field=tok.begin();
+    time_strings[1] = *field;
+    if (! is_valid_time(time_strings[1]) ) return 1;
+    field++;
+    int i=0;
+    for(; field!=tok.end();++field) {
+        i++;
+        if ( i%2 != 0 ) {
+            tmp_key.first = *field;
+            continue;
+        }
+        
+        t_tokenizer tok1(*field, sep1);
+        int j=0;
+        for(t_tokenizer::iterator field1=tok1.begin(); field1 != tok1.end();++field1) {
+            j++;
+            if ( j%2 != 0 ) {
+                tmp_key.second = *field1;
+                continue;
+            }
+
+            int tmp_value;
+            stringstream(*field1) >> tmp_value;
+            counts[tmp_key] = tmp_value;
+        }
+    }
+    return 0;
+
+}
+
+int
+Format2DSCStrategy::write_db(const DSCUMap &my_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    DSCUMap::const_iterator it1=my_dscdata.begin();
+    stringstream sql;
+    sql.clear();
+    int plot_id = get_plot_id(plot_name_, pg_db_trans);
+    sql.str("");
+
+    sql << "INSERT INTO dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\" (server_id, node_id, plot_id, starttime, key1, key2, value) values";
+
+    if (my_dscdata.empty()) {
+      cerr << "Format 2 Error: Empty data set for " << plot_name_ << " " << dtime[1] << ". Skipping..." << endl;
+      return 1;
+    }
+
+    while ( 1 ) {
+        if ( (*it1).second != 0 ) {
+            sql << " (" << server_id << ", " << node_id << ", " << plot_id
+                << ", timestamptz 'epoch' + " << dtime[1] <<  " * interval '1 second', '"
+                << (*it1).first.first << "','" << (*it1).first.second << "', " << (*it1).second << ")";
+
+            it1++;
+            if (it1 == my_dscdata.end() ) {
+                sql << ";";
+                break;
+            } else {
+                sql << ",";
+            }
+        } else {
+            cerr << "Format 2 Error: Bad data at " << plot_name_ << " " << dtime[1] << ". Skipping..." << endl;
+            return 1;
+        }
+    }
+
+    try {
+        pg_db_trans.exec(sql.str());
+    }
+    catch ( pqxx::unique_violation & e )
+    {
+        cerr << "Format 2 unique_violation error: " << e.what() << endl;
+        return 1;
+    } 
+    catch( runtime_error & e )
+    {
+        cerr << "Format 2 runtime error: " << e.what() << endl;
+        return 1;
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Format 2 exception: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Format 2 Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+    return 0;
+
+}
\ No newline at end of file
diff --git a/src/Format2DSCStrategy.h b/src/Format2DSCStrategy.h
new file mode 100644
index 0000000..139e146
--- /dev/null
+++ b/src/Format2DSCStrategy.h
@@ -0,0 +1,44 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#ifndef FORMAT2DSCSTRATEGY_H
+#define FORMAT2DSCSTRATEGY_H
+#include <string>
+#include <map>
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "pqxx/pqxx"
+
+using namespace std;
+
+class Format2DSCStrategy: public DSCStrategy {
+    
+public:
+
+    Format2DSCStrategy(string server, string name, string keys[], int keylength);
+    virtual ~Format2DSCStrategy() {};
+    void process_to_dat(const DSCUMap &, string dtime[]);
+    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line);
+    void process_dat_file(DSCUMap &/*counts*/, std::string /*time_strings*/[], const boost::filesystem::path &/*file_path*/) {};
+    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
+    bool is_dat_file_multi_unit() { return true; };
+
+private:
+
+    void write_dat(const DSCUMap &, string dtime[]);
+    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+
+};
+
+#endif	/* FORMAT2DSCSTRATEGY_H */
+
diff --git a/src/Format3aDSCStrategy.cpp b/src/Format3aDSCStrategy.cpp
new file mode 100644
index 0000000..4ee87a6
--- /dev/null
+++ b/src/Format3aDSCStrategy.cpp
@@ -0,0 +1,185 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+
+#include <iostream>
+#include <map>
+#include <string>
+
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/algorithm/string.hpp>
+
+#include "pqxx/pqxx"
+
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "Format3aDSCStrategy.h"
+
+//  Format3 has format k N(k)
+//  it also contains data for all time not just 1 day
+
+using namespace std;
+
+Format3aDSCStrategy::Format3aDSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
+    
+}
+
+void
+Format3aDSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string /*dtime*/[], int /*server*/, int /*node*/, pqxx::work* /*pg_db_trans*/) {
+	
+    // Only called if the input channel was XML
+    DSCKey tmp_key;
+    tmp_key.first = "All";
+    tmp_key.second = "All";
+    new_dscdata[tmp_key] = accum1d_to_count(orig_dscdata);
+
+}
+
+void
+Format3aDSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {
+    
+    open_file(dtime, ios_base::app | ios_base::out);
+    write_dat(dscdata, dtime);
+    close_file();
+
+}
+
+int
+Format3aDSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    return write_db(dscdata,dtime,server_id,node_id,pg_db_trans);
+
+}
+
+void
+Format3aDSCStrategy::write_dat(const DSCUMap &dscdata, string dtime[]) {
+    
+    DSCUMap::const_iterator it = dscdata.begin();
+    datafile_.filestream << dtime[1];
+    datafile_.filestream << " " << (*it).second << endl;
+
+}
+
+int
+Format3aDSCStrategy::process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) {
+
+    int tmp_value;
+    DSCKey tmp_key;
+    tmp_key.first = "All";
+    vector<string> splitvec;
+    vector<string>::iterator it;
+    
+    boost::split(splitvec, dat_line, boost::is_any_of(" "));
+    if ( splitvec.size() != 2 ) {
+        cerr << endl << "Found line containing: " << dat_line << " - skipped." << endl;
+        return 1;
+    }
+    it = splitvec.begin();
+    if(*it == "#MD5") return 1;
+    tmp_key.second = (*it);
+    time_strings[1] = (*it);
+    if (! is_valid_time(time_strings[1]) ) return 1;
+    it++;
+    stringstream((*it)) >> tmp_value;
+    
+    counts[tmp_key] = tmp_value;
+    splitvec.clear();
+    return 0;
+
+}
+
+void
+Format3aDSCStrategy::read_db(DSCUMap &old_dscdata, string dtime[],
+                     int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    DSCKey tmp_key;
+    tmp_key.first = "All";
+    int tmp_int = 0;
+    string tmp_str;
+    string key1 ("key1");
+    string key2 ("key2");
+    string value ("value");
+    stringstream sql;
+    int plot_id = get_plot_id(plot_name_, pg_db_trans);
+    sql << "SELECT key1, value FROM dsc.format1_3 where starttime=to_timestamp('" << dtime[1] 
+        << "') AND plot_id = '" << plot_id << "'" 
+        << " AND node_id = '" << node_id << "'"
+        << " AND server_id = '" << server_id << "';" << endl;
+
+    try{
+
+        const pqxx::result R = pg_db_trans.exec(sql.str());
+        pqxx::result::const_iterator row;
+        for (row = R.begin(); row != R.end(); ++row) {
+            row[key1].to(tmp_str);
+            row[value].to(tmp_int);
+            tmp_key.second = tmp_str;
+            old_dscdata[tmp_key] = tmp_int;
+        }
+    }
+    catch( runtime_error & e )
+    {
+        cerr << "Format 3a runtime error: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Format 3a exception: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Format 3a Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+}
+
+int
+Format3aDSCStrategy::write_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    stringstream sql;
+    int plot_id = get_plot_id(plot_name_, pg_db_trans);
+    DSCUMap::const_iterator it = dscdata.begin();    
+    sql << "INSERT INTO dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2]
+        << "\" (server_id, node_id, plot_id, starttime, key1, key2, value) values ("
+        << server_id << ", " << node_id << ", " << plot_id
+        << ", timestamptz 'epoch' + " << dtime[1] <<  " * interval '1 second', '" 
+        << dtime[1] << "', 'NONE', " << (*it).second << ");" << endl;
+    try {
+        (void)pg_db_trans.exec(sql.str());
+    }
+    
+    catch ( pqxx::unique_violation & e )
+    {
+        cerr << "Format 3a unique_violation error: " << e.what() << endl;
+        return 1;
+    }   
+    catch( runtime_error & e )
+    {
+        cerr << "Format 3a runtime error: " << e.what() << endl;
+        return 1;
+    }
+    catch( std::exception & e )
+    {
+        cerr << "Format 3a exception: " << e.what() << endl;
+        exit( EXIT_FAILURE );
+    }
+    catch( ... )
+    {
+        cerr << "Format 3a Unknown exception caught" << endl;
+        exit( EXIT_FAILURE );
+    }
+    return 0;
+
+}
diff --git a/src/Format3aDSCStrategy.h b/src/Format3aDSCStrategy.h
new file mode 100644
index 0000000..6e48d3a
--- /dev/null
+++ b/src/Format3aDSCStrategy.h
@@ -0,0 +1,50 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   Format3DSCStrategy.h
+ */
+
+#ifndef FORMAT3aDSCSTRATEGY_H
+#define FORMAT3aDSCSTRATEGY_H
+#include <string>
+#include <map>
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "pqxx/pqxx"
+
+using namespace std;
+
+class Format3aDSCStrategy: public DSCStrategy {
+    
+public:
+
+    Format3aDSCStrategy(string server, string name, string keys[], int keylength);
+    virtual ~Format3aDSCStrategy() {};
+
+    void process_to_dat(const DSCUMap &, string dtime[]);
+    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line);
+    void process_dat_file(DSCUMap &/*counts*/, std::string /*time_strings*/[], const boost::filesystem::path &/*file_path*/) {};
+    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
+    bool is_dat_file_multi_unit() { return true; };
+
+private:
+
+    void write_dat(const DSCUMap &dscdata, string dtime[]);    
+    int write_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    void read_db(DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+
+};
+
+#endif	/* FORMAT3aDSCSTRATEGY_H */
+
diff --git a/src/Format3bDSCStrategy.cpp b/src/Format3bDSCStrategy.cpp
new file mode 100644
index 0000000..591da0e
--- /dev/null
+++ b/src/Format3bDSCStrategy.cpp
@@ -0,0 +1,169 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+
+#include <iostream>
+#include <map>
+#include <string>
+
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/algorithm/string.hpp>
+
+#include "pqxx/pqxx"
+
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "Format3bDSCStrategy.h"
+
+// Format3b has format k N(k)
+
+using namespace std;
+
+Format3bDSCStrategy::Format3bDSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
+    
+}
+
+void
+Format3bDSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string dtime[], int /*server_id*/, int /*node_id*/, pqxx::work* pg_db_trans) {
+    
+    if (pg_db_trans == NULL) {
+        open_file(dtime, ios_base::in);
+        read_dat(new_dscdata);
+        merge_accum1d(orig_dscdata,new_dscdata);
+        close_file();
+    }
+    else {
+        new_dscdata = orig_dscdata;
+    }
+
+}
+
+void
+Format3bDSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {
+
+    // reopen dat file with trunc to write out merged data
+    open_file(dtime, ios_base::trunc | ios_base::out);
+    write_dat(dscdata,dtime);
+    close_file();
+
+}
+
+int
+Format3bDSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    return write_db(dscdata,dtime,server_id,node_id,pg_db_trans);
+
+}
+
+void
+Format3bDSCStrategy::write_dat(const DSCUMap &old_dscdata, string /*dtime*/[]) {
+
+    DSCUMap::const_iterator it = old_dscdata.begin();
+    for ( ; it != old_dscdata.end() ; it++) {
+        datafile_.filestream << (*it).first.second << " " << (*it).second << endl;
+    }
+
+}
+
+void
+Format3bDSCStrategy::read_dat(DSCUMap &old_dscdata) {
+
+    int tmp_value;
+    DSCKey tmp_key;
+    tmp_key.first = "All";
+    string line;
+    vector<string> splitvec;
+    vector<string>::iterator it;
+    while (getline(datafile_.filestream,line)) {
+        boost::split(splitvec, line, boost::is_any_of(" "));
+        if ( splitvec.size() != 2 ) {
+            cerr << "Bad data skipped." << endl;
+            continue;
+        }
+        it = splitvec.begin();
+        tmp_key.second = (*it);
+        if (tmp_key.second == "#MD5") continue;
+        it++;
+        stringstream((*it)) >> tmp_value;
+        old_dscdata[tmp_key] = tmp_value;
+        splitvec.clear();
+    }
+
+}
+
+void
+Format3bDSCStrategy::process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &/*file_path*/) {
+
+    open_file(time_strings, ios_base::in);
+    read_dat(counts);
+    close_file();
+
+}
+
+int
+Format3bDSCStrategy::write_db(const DSCUMap &old_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    DSCUMap::const_iterator it1 = old_dscdata.begin();
+    stringstream sql;
+    stringstream tablename;
+    sql.clear();
+    int plot_id = get_plot_id(plot_name_, pg_db_trans);
+    sql.str("");
+    tablename << "dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\"";
+
+    for ( ; it1 != old_dscdata.end() ; it1++ ) {    
+        sql.clear();
+        sql.str("");
+        sql << "with upsert as ( update " << tablename.str() << " as d "
+                << " set value= " << (*it1).second << " + e.value "
+                << " from " << tablename.str() << " as e "
+                << " where d.starttime=e.starttime and d.server_id=e.server_id and d.node_id=e.node_id and d.plot_id=e.plot_id and d.key1=e.key1 and d.key2=e.key2 "
+                << " and e.server_id= " << server_id
+                << " and e.node_id= " << node_id 
+                << " and e.plot_id= " << plot_id 
+                << " and e.starttime= " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')"
+                << " and e.key1= '" << (*it1).first.first << "'"
+                << " and e.key2= '" << (*it1).first.second  << "'"
+                << " returning d.* )"
+                << " insert into " << tablename.str()
+                << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
+                << " select " << server_id << " , " << node_id << " , " << plot_id << " , " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')" << " , '" << (*it1).first.first << "', '"  << (*it1).first.second << "', " << (*it1).second
+                << " where not exists (select 1 from upsert);";
+        try {       
+            pg_db_trans.exec(sql.str());
+        }
+        catch ( pqxx::unique_violation & e )
+        {
+            cerr << "Format 3b unique_violation error: " << e.what() << endl;
+            return 1;
+        }   
+        catch( runtime_error & e )
+        {
+            cerr << "Format 3b write_db runtime error: " << e.what() << endl;
+            return 1;
+        }
+        catch( std::exception & e )
+        {
+            cerr << "Format 3b write_db exception: " << e.what() << endl;
+            exit( EXIT_FAILURE );
+        }
+        catch( ... )
+        {
+            cerr << "Format 3b write_db Unknown exception caught" << endl;
+            exit( EXIT_FAILURE );
+        }
+    }
+    return 0;
+
+}
diff --git a/src/Format3bDSCStrategy.h b/src/Format3bDSCStrategy.h
new file mode 100644
index 0000000..2382de3
--- /dev/null
+++ b/src/Format3bDSCStrategy.h
@@ -0,0 +1,52 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   Format3DSCStrategy.h
+ */
+
+#ifndef FORMAT3bDSCSTRATEGY_H
+#define FORMAT3bDSCSTRATEGY_H
+#include <string>
+#include <map>
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "pqxx/pqxx"
+
+using namespace std;
+
+class Format3bDSCStrategy: public DSCStrategy {
+    
+public:
+
+    Format3bDSCStrategy(string server, string name, string keys[], int keylength);
+    virtual ~Format3bDSCStrategy() {};
+
+    void process_to_dat(const DSCUMap &, string dtime[]);
+    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    int process_dat_line(DSCUMap &/*counts*/, std::string /*time_strings*/[], std::string /*dat_line*/) { return 0; };
+    void process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &file_path);
+    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
+    bool is_dat_file_multi_unit() { return false; };
+
+private:
+
+    void write_dat(const DSCUMap &, string dtime[]);
+    void write_count_dat(int count, string dtime[]);    
+    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    void write_count_db(int count, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    void read_dat(DSCUMap &);
+
+};
+
+#endif	/* FORMAT3bDSCSTRATEGY_H */
+
diff --git a/src/Format4DSCStrategy.cpp b/src/Format4DSCStrategy.cpp
new file mode 100644
index 0000000..9d2cf02
--- /dev/null
+++ b/src/Format4DSCStrategy.cpp
@@ -0,0 +1,366 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+#include <iostream>
+#include <map>
+#include <string>
+#include <vector>
+
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/algorithm/string.hpp>
+#include <boost/foreach.hpp>
+
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "Format4DSCStrategy.h"
+
+#include "pqxx/pqxx"
+
+// Format4 has format j k N(j,k)
+
+using namespace std;
+namespace bfs = boost::filesystem;
+
+Format4DSCStrategy::Format4DSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
+
+}
+
+void
+Format4DSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string dtime[], int /*server_id*/, int /*node_id*/, pqxx::work* pg_db_trans) {
+
+    DSCUMap tmp_dscdata;
+
+    // Call the munger
+    // It depends on m_name - the if may be better in the factory.
+    if ( plot_name_ == "client_subnet2_accum" || plot_name_ == "qtype_vs_tld" || plot_name_ == "client_addr_vs_rcode_accum") {
+        swap_dimensions(orig_dscdata, tmp_dscdata);
+    } else if ( plot_name_ == "dns_ip_version_vs_qtype" ) {
+        tmp_dscdata = orig_dscdata;
+        elsify_unwanted_keys(tmp_dscdata);  
+    } else {
+        tmp_dscdata = orig_dscdata;
+    }
+    
+    // If we are writing to a dat file then we need to merge data
+    // If we are writing to the database then we will use a CTE upsert
+    if (pg_db_trans == NULL) {
+        open_file(dtime, ios_base::in);   
+        // Read the old dat file in as we need to merge old and new data
+        read_dat(new_dscdata);   
+        close_file();
+        merge_accum2d(tmp_dscdata, new_dscdata);
+        // Trimmer: runs once per hour of xml data
+        if ( plot_name_ == "client_subnet2_accum" || plot_name_ == "qtype_vs_tld" || plot_name_ == "client_subnet_vs_tld") {
+                int tmp_value;
+                stringstream(dtime[1]) >> tmp_value;
+                if ((59*60) == (tmp_value % 3600)) {
+                        trim_accum2d(new_dscdata);
+                }
+        } 
+    }  else {
+        new_dscdata = tmp_dscdata;
+    }
+    
+}
+
+void
+Format4DSCStrategy::process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &/*file_path*/) {
+
+    open_file(time_strings, ios_base::in);
+    read_dat(counts);
+    close_file();
+
+}
+
+void
+Format4DSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {
+
+    open_file(dtime, ios_base::trunc | ios_base::out);
+    write_dat(dscdata,dtime);
+    close_file();
+
+}
+
+int
+Format4DSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {
+
+    return write_db(dscdata,dtime, server_id, node_id, pg_db_trans);
+
+}
+
+void
+Format4DSCStrategy::write_dat(const DSCUMap &old_dscdata, string /*dtime*/[]){
+
+    DSCUMap::const_iterator it1=old_dscdata.begin();
+    for ( ; it1 != old_dscdata.end() ; it1++ ) {
+        datafile_.filestream << (*it1).first.first << " " << (*it1).first.second << " " << (*it1).second << endl;
+    }
+
+}
+
+void
+Format4DSCStrategy::read_dat(DSCUMap &old_dscdata) {
+
+    string token;
+    int tmp_value;
+    DSCKey tmp_key;
+    string line;
+    vector<string> splitvec;
+    vector<string>::iterator it;
+    while (getline(datafile_.filestream,line)) {
+        
+        stringstream lineStream(line);
+        
+        lineStream >> tmp_key.first;
+        if (tmp_key.first == "#MD5") continue;
+        lineStream >> tmp_key.second;
+        lineStream >> tmp_value;
+        if (! lineStream.eof()) {
+            cerr << "Bad data reading" << plot_name_ << "dat file" << endl;
+            continue;
+        }
+        old_dscdata[tmp_key] = tmp_value;
+    }
+
+}
+
+int
+Format4DSCStrategy::write_db(const DSCUMap &old_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans){
+
+    stringstream sql;
+    stringstream tablename;
+    DSCUMap::const_iterator it1=old_dscdata.begin();
+    sql.clear();
+    int plot_id = get_plot_id(plot_name_, pg_db_trans);
+    sql.str("");    
+    tablename << "dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\"";
+    for ( ; it1 != old_dscdata.end() ; it1++ ) {    
+        sql.clear();
+        sql.str("");
+        sql << "with upsert as ( update " << tablename.str() << " as d "
+                << " set value= " << (*it1).second << " + e.value "
+                << " from " << tablename.str() << " as e "
+                << " where d.starttime=e.starttime and d.server_id=e.server_id and d.node_id=e.node_id and d.plot_id=e.plot_id and d.key1=e.key1 and d.key2=e.key2 "
+                << " and e.server_id= " << server_id
+                << " and e.node_id= " << node_id 
+                << " and e.plot_id= " << plot_id 
+                << " and e.starttime= " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')"
+                << " and e.key1= '" << (*it1).first.first << "'"
+                << " and e.key2= '" << (*it1).first.second  << "'"
+                << " returning d.* )"
+                << " insert into " << tablename.str()
+                << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
+                << " select " << server_id << " , " << node_id << " , " << plot_id << " , " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')" << " , '" << (*it1).first.first << "', '"  << (*it1).first.second << "', " << (*it1).second
+                << " where not exists (select 1 from upsert);";    
+        try {
+            pg_db_trans.exec(sql.str());
+        }
+        catch ( pqxx::unique_violation & e ) {
+            cerr << "Format 4 unique_violation error: " << e.what() << endl;
+            return 1;
+        }
+        catch( runtime_error & e )
+        {
+            cerr << "Format 4 write_db Runtime error: " << e.what() << endl;
+            return 1;
+        }
+        catch( std::exception & e )
+        {
+            cerr << "Format 4 write_db exception: " << e.what() << endl;
+            exit( EXIT_FAILURE );
+        }
+        catch( ... )
+        {
+            cerr << "Format 4 write_db Unknown exception caught" << endl;
+            exit( EXIT_FAILURE );
+        }
+    }
+    
+    // Trimmer: runs once per hour of xml data
+    int debug = 0;
+    int tmp_value;
+    stringstream(dtime[1]) >> tmp_value;
+    if ((59*60) == (tmp_value % 3600)) {
+        if ( debug == 1 ) cout << "Trimmer Started." << endl;
+        if ( plot_name_ == "client_subnet2_accum" || plot_name_ == "qtype_vs_tld" || plot_name_ == "client_subnet_vs_tld") {
+            if ( debug == 1 ) cout << "Trimmer found plot " << plot_name_ << endl;
+            
+            
+            // first create a string to hold all the common query parameters
+            stringstream sql_query_params;
+            sql_query_params << " and server_id= " << server_id
+                             << " and node_id= " << node_id 
+                             << " and plot_id= " << plot_id 
+                             << " and starttime= " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD') ";
+            stringstream sql_ignore_skipped;
+            sql_ignore_skipped << " key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') ";
+                  
+            // Now figure out all the key2s which need trimming and loop over them
+            string key2totrim;
+            string key2 ("key2");
+            sql.clear();
+            sql.str("");
+            sql << "select key2 from (select key2, count(*) from " << tablename.str()
+                << " where " << sql_ignore_skipped.str() << sql_query_params.str()
+                << " group by key2) as to_trim where count > 1000;";
+            if ( debug == 1 ) cout << "Trimmer executed: " << sql.str() << endl << endl;
+            try {
+                 const pqxx::result R = pg_db_trans.exec(sql.str());
+                 if ( debug == 1 ) cout << "Trimmer got back " << R.size() << " rows." << endl << endl;
+                 pqxx::result::const_iterator row;
+                 for (row = R.begin(); row != R.end(); ++row) {
+                     row[key2].to(key2totrim);
+                     stringstream sql1, sql2, sql3, sql4, sql5, sql6;
+                     string skipped_str ("skipped");
+                     string skipped_sum_str ("skipped_sum");                 
+                     int skipped = 0;
+                     int skipped_sum = 0;
+                     int skipped_new = 0;
+                     int skipped_sum_new = 0;
+                                                          
+                     // Calculate the new values - i.e. row count above 1000    
+                     sql1 << "select sum(value) as skipped_sum, count(value) as skipped"
+                          << " from "
+                                << " (select key1, value from " << tablename.str()
+                                << " where key2= '" << key2totrim << "' "
+                                << " and " << sql_ignore_skipped.str() << sql_query_params.str() 
+                                << " order by value desc offset 1000) as rowstotrim;";
+                     if ( debug == 1 ) cout << "Trimmer executed: " << sql1.str() << endl << endl;
+                     const pqxx::result R1 = pg_db_trans.exec(sql1.str());
+                     if ( debug == 1 ) cout << "Trimmer got back " << R1.size() << " rows." << endl << endl;
+                     pqxx::result::const_iterator row1;
+                     for (row1 = R1.begin(); row1 != R1.end(); ++row1) {
+                         row1[skipped_str].to(skipped_new);
+                         row1[skipped_sum_str].to(skipped_sum_new);
+                     }
+                     if ( debug == 1 ) cout << "Trimmer skipped_new = " << skipped_new << ", skipped_sum_new = " << skipped_sum_new << endl << endl;
+                     
+                     // Pull the existing values out of the database for skipped and skipped sum
+                     // and update or insert the new ones
+                     sql2 << "select value as skipped_sum from " << tablename.str() 
+                          << " where key2= '" << key2totrim 
+                          << "' and key1='-:SKIPPED_SUM:-'"
+                          << sql_query_params.str() << ";";
+                     if ( debug == 1 ) cout << "Trimmer executed: " << sql2.str() << endl << endl;
+                     const pqxx::result R2 = pg_db_trans.exec(sql2.str());
+                     if ( debug == 1 ) cout << "Trimmer got back " << R2.size() << " rows." << endl << endl;
+                     if (R2.size() > 1 ) {
+                        cerr << "Format 4 write_db Expected 0 or 1 skipped_sum records. Found " << R2.size() << endl;
+                        exit( EXIT_FAILURE );                     
+                     }                    
+                     if (R2.size() == 1) {
+                         R2.begin()[skipped_sum_str].to(skipped_sum);
+                         if ( debug == 1 ) cout << "Trimmer skipped_sum existing= " << skipped_sum << endl << endl;
+                         skipped_sum = skipped_sum + skipped_sum_new;
+                         if ( debug == 1 ) cout << "Trimmer skipped_sum updated= " << skipped_sum << endl << endl;
+                         // Now update them in the database
+                         sql4 << "update " << tablename.str()
+                              << " set value = " << skipped_sum 
+                              << " where key2='" << key2totrim << "' "
+                              << " and key1='-:SKIPPED_SUM:-'"
+                              << sql_query_params.str() << ";";
+                         if ( debug == 1 ) cout << "Trimmer executed: " << sql4.str() << endl << endl;
+                     }
+                     else {
+                        sql4 << " insert into " << tablename.str()
+                             << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
+                             << " values (" << server_id << " , " 
+                             << node_id << " , " 
+                             << plot_id << " , " 
+                             << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD') " 
+                             << ", '-:SKIPPED_SUM:-', '"  
+                             << key2totrim << "', " 
+                             << skipped_sum_new << ");";  
+                        if ( debug == 1 ) cout << "Trimmer executed: " << sql4.str() << endl << endl;
+                     }
+                     const pqxx::result R10 = pg_db_trans.exec(sql4.str());
+                     if ( debug == 1 ) cout << "Trimmer affected " << R10.affected_rows() << " rows." << endl << endl;
+                     sql3 << "select value as skipped from " << tablename.str() 
+                          << " where key2= '" << key2totrim 
+                          << "' and key1='-:SKIPPED:-'"
+                          << sql_query_params.str() << ";";
+                     if ( debug == 1 ) cout << "Trimmer executed: " << sql3.str() << endl << endl;
+                     const pqxx::result R3 = pg_db_trans.exec(sql3.str());
+                     if ( debug == 1 ) cout << "Trimmer got back " << R3.size() << " rows." << endl << endl;
+                     pqxx::result::const_iterator row3;
+                     if (R3.size() > 1 ) {
+                        cerr << "Format 4 write_db Expected 0 or 1 skipped records. Found " << R3.size() << endl;
+                        exit( EXIT_FAILURE );                     
+                     }                    
+                     if (R3.size() == 1) {
+                         R3.begin()[skipped_str].to(skipped);
+                         if ( debug == 1 ) cout << "Trimmer skipped existing = " << skipped << endl << endl;
+                         skipped = skipped + skipped_new;
+                         if ( debug == 1 ) cout << "Trimmer skipped updated= " << skipped << endl << endl;
+                         sql5 << "update " << tablename.str()
+                              << " set value = " << skipped 
+                              << " where key2='" << key2totrim << "' "
+                              << " and key1='-:SKIPPED:-'"
+                              << sql_query_params.str() << ";";
+                         if ( debug == 1 ) cout << "Trimmer executed: " << sql5.str() << endl << endl;
+                     }
+                     else {
+                         sql5 << " insert into " << tablename.str()
+                             << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
+                             << " values (" << server_id << " , " 
+                             << node_id << " , " 
+                             << plot_id << " , " 
+                             << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD') " 
+                             << ", '-:SKIPPED:-', '"  
+                             << key2totrim << "', " 
+                             << skipped_new << ");";             
+                         if ( debug == 1 ) cout << "Trimmer executed: " << sql5.str() << endl << endl;
+                     }                           
+                     const pqxx::result R11 = pg_db_trans.exec(sql5.str());
+                     if ( debug == 1 ) cout << "Trimmer affected " << R11.affected_rows() << " rows." << endl << endl;
+
+                     // Now we need to delete all the skipped keys
+                     sql6 << "delete from " << tablename.str() 
+                          << "where key2='" << key2totrim << "' "
+                          << sql_query_params.str()
+                          << " and key1 in "
+                                << " (select key1 from " << tablename.str()
+                                << " where key2= '" << key2totrim << "' "
+                                << " and  " << sql_ignore_skipped.str() << sql_query_params.str() 
+                                << " order by value desc offset 1000);";
+                     if ( debug == 1 ) cout << "Trimmer executed: " << sql6.str() << endl << endl;
+                     
+                     const pqxx::result R12 = pg_db_trans.exec(sql6.str());
+                     if ( debug == 1 ) cout << "Trimmer affected " << R12.affected_rows() << " rows." << endl << endl;
+                }                  
+            }
+            catch ( pqxx::unique_violation & e ) {
+                cerr << "Format 4 unique_violation error: " << e.what() << endl;
+                return 1;
+            }
+            catch( runtime_error & e )
+            {
+                cerr << "Format 4 write_db Runtime error: " << e.what() << endl;
+                return 1;
+            }
+            catch( std::exception & e )
+            {
+                cerr << "Format 4 write_db exception: " << e.what() << endl;
+                exit( EXIT_FAILURE );
+            }
+            catch( ... )
+            {
+                cerr << "Format 4 write_db Unknown exception caught" << endl;
+                exit( EXIT_FAILURE );
+            }
+        } 
+    }
+    return 0;
+
+}
\ No newline at end of file
diff --git a/src/Format4DSCStrategy.h b/src/Format4DSCStrategy.h
new file mode 100644
index 0000000..c7621c9
--- /dev/null
+++ b/src/Format4DSCStrategy.h
@@ -0,0 +1,51 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   Format4DSCStrategy.h
+ */
+
+#ifndef FORMAT4DSCSTRATEGY_H
+#define FORMAT4DSCSTRATEGY_H
+
+#include <string>
+#include <map>
+#include "dsc_types.h"
+#include "DSCStrategy.h"
+#include "pqxx/pqxx"
+
+using namespace std;
+
+class Format4DSCStrategy: public DSCStrategy {
+
+public:
+
+    Format4DSCStrategy(string server, string name, string keys[], int keylength);
+    virtual ~Format4DSCStrategy() {};
+
+    void process_to_dat(const DSCUMap &, string dtime[]);
+    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    int process_dat_line(DSCUMap &/*counts*/, std::string /*time_strings*/[], std::string /*dat_line*/) { return 0; };
+    void process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &file_path);
+    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
+    bool is_dat_file_multi_unit() { return false; };
+
+private:
+	
+    void write_dat(const DSCUMap &, string dtime[]);
+    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
+    void read_dat(DSCUMap &);
+
+};
+
+#endif	/* FORMAT4DSCSTRATEGY_H */
+
diff --git a/src/Makefile.am b/src/Makefile.am
new file mode 100644
index 0000000..b1e9ef1
--- /dev/null
+++ b/src/Makefile.am
@@ -0,0 +1,78 @@
+if DM
+D = dsc-extractor
+P = rssac_server.py rssac_server/rssac.py rssac_server/threadpool.py
+S = rssacd
+else
+D =
+P =
+S =
+endif
+
+hhconfdir = @sysconfdir@/hedgehog
+
+bin_PROGRAMS = $(D)
+
+dsc_extractor_SOURCES = dsc-extractor.cpp \
+                          DSCDataManager.cpp \
+                          DSCIOManager.cpp \
+                          DSCStrategyFactory.cpp \
+                          DSCStrategy.cpp \
+                          Format1DSCStrategy.cpp \
+                          Format2DSCStrategy.cpp \
+                          Format3aDSCStrategy.cpp \
+                          Format3bDSCStrategy.cpp \
+                          Format4DSCStrategy.cpp \
+                          DSCDataManager.h \
+                          DSCStrategyFactory.h \
+                          DSCStrategy.h \
+                          DSCIOManager.h \
+                          Format1DSCStrategy.h \
+                          Format2DSCStrategy.h \
+                          Format3aDSCStrategy.h \
+                          Format3bDSCStrategy.h \
+                          Format4DSCStrategy.h \
+                          dsc_types.h
+
+dsc_extractor_CXXFLAGS = @LIBPQXX_INCLUDES@ @BOOST_CPPFLAGS@ -I$(top_srcdir)/src/libdsc
+
+dsc_extractor_LDADD = @BOOST_SYSTEM_LDFLAGS@ \
+                          @BOOST_SYSTEM_LIBS@ \
+                          @BOOST_FILESYSTEM_LDFLAGS@ \
+                          @BOOST_FILESYSTEM_LIBS@ \
+                          @BOOST_PROGRAM_OPTIONS_LDFLAGS@ \
+                          @BOOST_PROGRAM_OPTIONS_LIBS@ \
+                          @LIBPQXX_LIBS@
+
+python_PYTHON = $(P)
+sbin_SCRIPTS = $(S)
+EXTRA_DIST = rssac_server.in rssacd.in
+
+do_subst_hh = sed -e 's|@HEDGEHOGYAML@|$(hhconfdir)|' -e "s|@DB_VERSION@|@DB_SCHEMA_VERSION@|"
+do_subst_python = sed -e 's|@PYTHON_EXE@|$(PYTHON)|' -e 's|@HHPYSITEDIR@|$(pythondir)|'
+
+rssac_server.py: $(srcdir)/rssac_server/rssac_server.in Makefile
+	$(do_subst_hh) < $(srcdir)/rssac_server/rssac_server.in > rssac_server.py
+	
+rssacd: $(srcdir)/rssac_server/rssacd.in Makefile
+	$(do_subst_python) < $(srcdir)/rssac_server/rssacd.in > rssacd
+
+dist_man_MANS = rssacd.8 \
+                dsc-extractor.1
+
+rssacd.8: rssacd rssac_server.py rssac_server/rssac.py rssac_server/threadpool.py Makefile
+	chmod 755 rssacd
+	PYTHONPATH=${srcdir}/rssac_server $(HELP2MAN) --version-string=@PACKAGE_VERSION@ -N -s 8 -n "This listens for DNS Notify messages and records the zone size and the time for propogation of zones. This is only needed for the generation of RSSAC reports" ./rssacd -o rssacd.8
+
+dsc-extractor.1: dsc-extractor Makefile
+	$(HELP2MAN) -h -h --version-string=@PACKAGE_VERSION@ -N -s 1 --no-discard-stderr -n "Application to extract DSC data from XML or DAT files and insert it in to a Postgresql database. Users are recommended to use the refile_and_grok wrapper script in preference to using this exectutable directly." ./dsc-extractor -o dsc-extractor.1
+
+dist_sysconf_DATA = rssacd.upstart \
+                    rssac_server/rssacd.conf
+
+rssacd.upstart: $(srcdir)/rssac_server/rssacd.upstart.in Makefile
+	sed -e 's|@HEDGEHOGYAML@|$(hhconfdir)|' -e 's|@SBIN@|$(sbindir)|' < $(srcdir)/rssac_server/rssacd.upstart.in > rssacd.upstart
+
+install-data-hook:
+	mkdir -p ${DESTDIR}${sysconfdir}/init
+	mv ${DESTDIR}${sysconfdir}/rssacd.upstart ${DESTDIR}${sysconfdir}/init/rssacd.conf
+	mv ${DESTDIR}${sysconfdir}/rssacd.conf ${DESTDIR}${hhconfdir}/rssacd.conf
diff --git a/src/dsc-extractor.cpp b/src/dsc-extractor.cpp
new file mode 100644
index 0000000..3e11178
--- /dev/null
+++ b/src/dsc-extractor.cpp
@@ -0,0 +1,178 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/// Application which can insert data from DSC XML and DAT files into
+/// a PostgreSQL database. It can also convert from XML to DAT.
+
+#include "config.h"
+#include <iostream>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+
+#include "dsc_types.h"
+#include "DSCIOManager.h"
+
+#define LOCK_FILE       "dsc-extractor.lock"
+
+using namespace std;
+
+/// \brief Usage function that prints usage information.
+static void
+usage (void)
+{
+    fprintf(stderr, "Usage: dsc-extractor [OPTION]...\n");
+    fprintf(stderr,
+        "Supported options:\n"
+                "  -h                   Show this usage. \n"
+                "  -i                   Input source: [XML|DAT]. Default XML. \n"
+                "  -o                   Output destination: [DAT|PG_DB]. Default DAT. \n"
+                "  -c                   Connection string for DB. Required. \n"
+                "  -s                   Start date for data file processing. Default is to process all data. \n"
+                "  -r                   Disable processing of rssac data. Default is to process all data. \n");
+    fprintf(stderr, "Version %s. Report bugs to <%s>.\n",
+            PACKAGE_VERSION, PACKAGE_BUGREPORT);  
+}
+
+/// \brief Main function parses command line and calls the DSCIOManager to load 
+/// and convert the data.
+int
+main(int argc, char *argv[]) {
+    int pfp;
+    char str[32];
+    pid_t pid;
+    pid = getpid();
+    pfp=open(LOCK_FILE, O_RDWR|O_CREAT, 0640);
+    if ( pfp < 0 ) {
+        cerr << "Error: Unable to open pid file " << LOCK_FILE << endl;
+        exit(1);
+    }
+    if ( lockf(pfp,F_TLOCK,0) < 0 ) {
+        cerr << "Error: Unable to lock pid file " << LOCK_FILE << endl;
+        close(pfp);
+        exit(0);
+    }
+    if ( snprintf(str, sizeof(str), "%lu\n", (unsigned long) pid) < 0 ) {
+        cerr << "Error: Unable to convert pid to string.\n" << endl;
+        close(pfp);
+        exit(1);
+    }
+    if ( write(pfp,str,strlen(str)) < 0 ) {
+        cerr << "Error: Unable to write to pid file " << LOCK_FILE << endl;
+        close(pfp);
+        exit(1);
+    }
+    
+    // default settings that make this code work as a drop in replacement for
+    // the original dsc-xml-extractor.pl 
+    enum INPUT_OUTPUT input  = XML;
+    enum INPUT_OUTPUT output = DAT;
+    // connection string to be used if the output is to a database
+    string conn_string;
+    // string used to determine which date to start processing files from
+    string processing_start_date_string;
+    // process rssac data by default if present
+   bool rssac=true; 
+    
+    // parse command line options
+    int c;
+    while ((c = getopt(argc, argv, "hi:o:c:s:r")) != -1) {
+        switch (c) {
+        case 'h':
+            usage();
+            exit(0);
+        case 'i': {
+            string input_string = optarg;
+            if (input_string.compare("XML") == 0) {
+                input=XML;
+            } else if (input_string.compare("DAT") == 0) {
+                input=DAT;
+            } else {
+                cout << "Error: Invalid -i value" << endl;
+                usage();
+                exit(1);
+            }
+            break;
+        }
+        case 'o': {        
+            string output_string = optarg;
+            if (output_string.compare("DAT") == 0) {
+                output=DAT;
+            } else if (output_string.compare("PG_DB") == 0) {
+                output=PG_DB;
+            } else {
+                cout << "Error: Invalid -o value" << endl;
+                usage();
+                exit(1);
+            }
+            break;
+        }    
+        case 'c': {
+            conn_string = optarg;
+            if (conn_string.empty()) {
+                cout << "Error: A connection string must be supplied with the -c flag." << endl;
+                usage();
+                exit(1);
+            }
+            break;
+        }
+        case 's': {
+            processing_start_date_string = optarg;
+            if (processing_start_date_string.empty()) {
+                cout << "A processing start date in format YYYY-MM-DD must be provided when using -s." << endl;
+                exit(1);
+            }
+            break;
+        }
+       case 'r': {
+            rssac=false;
+            cout << "RSSAC data in the XML will NOT be processed" << endl;
+            break;
+        }
+        case '?': {
+            usage();
+            exit(1);
+        }        
+        default: {
+            // nothing to do here just go with defaults
+            break;
+        }    
+        }
+    }
+    
+    if (input == output) {
+        cout << "Error: Input and output cannot be the same. \n";
+        exit(1);
+    }   
+    if ( input != XML && ! processing_start_date_string.empty()) {
+        cerr << "Error: Processing start date option is only supported for input type of XML. Exiting." << endl;
+        exit(1);
+    }    
+    
+    try {
+        DSCIOManager* data_io_manager = new DSCIOManager(input, output, conn_string, processing_start_date_string, rssac);
+        data_io_manager->dsc_activate_ouput_destination();
+        data_io_manager->dsc_import_input_from_source();
+        data_io_manager->dsc_deactivate_ouput_destination();
+        delete data_io_manager;
+    }
+    catch (std::exception &e) {
+        cerr << "Error in libdsc: " << e.what() << endl;
+        return 1;
+    }
+    
+    if ( unlink(LOCK_FILE) < 0 ) {
+        cerr << "Error: Unable to unlink pid file " << LOCK_FILE << endl;
+    }
+    return 0;
+}
diff --git a/src/dsc_types.h b/src/dsc_types.h
new file mode 100644
index 0000000..8ec47c6
--- /dev/null
+++ b/src/dsc_types.h
@@ -0,0 +1,77 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* 
+ * File:   dsc_types.h
+ */
+
+#ifndef DSC_TYPES_H
+#define	DSC_TYPES_H
+#define BOOST_FILESYSTEM_NO_DEPRECATED
+
+#include <string>
+
+#include <boost/filesystem.hpp>
+#include <boost/filesystem/fstream.hpp>
+#include <boost/unordered_map.hpp>
+
+#include <pqxx/pqxx>
+
+using namespace std;
+using namespace boost;
+namespace bfs = boost::filesystem;
+
+/// \brief Numeration to define valid input sources and output destinations.
+enum INPUT_OUTPUT {
+    XML,
+    DAT,
+    PG_DB
+};
+
+/// \brief Typedef defining the key pair that is used in the data map.
+typedef pair<string, string> DSCKey;
+
+/// \brief Ensure that there is a hash function for use in
+/// boost::unordered_map
+namespace boost {
+    template <>
+            class hash<DSCKey> {
+            public:
+            size_t operator() ( const DSCKey &akey ) const
+            {
+                size_t v=0;
+                boost::hash_combine(v, akey.first);
+                boost::hash_combine(v, akey.second);
+                return v;
+            }
+    };
+}
+
+/// \brief Typedef defining a map that can hold any type of DSC data.
+typedef boost::unordered_map<DSCKey,int> DSCUMap;
+
+/// \brief Structure to hold information about dat files that are being manipulated
+/// by the DSC code.
+struct dsc_dat_file {
+    string filename;
+    string tmpfilename;
+    bfs::fstream filestream;
+    ios_base::openmode open_mode;
+};
+typedef dsc_dat_file DSCdatfile;
+
+
+/// \brief Max number of strategies for a given dsc xml array name.
+/// 4 is fine for all the current ones.
+//#define MAXSTRATEGIES 4
+
+#endif	/* DSC_TYPES_H */
diff --git a/src/rssac_server/rssac.py b/src/rssac_server/rssac.py
new file mode 100755
index 0000000..33e6a84
--- /dev/null
+++ b/src/rssac_server/rssac.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python
+# 
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+#
+# Original developed by john.bond@icann.org
+# Modified for use with Hedgehog by Sinodun IT (www.sinodun.com)
+
+import argparse
+import os
+import glob
+import yaml
+import time
+import sys
+import dns.query
+import dns.message
+import dns.rdatatype
+import dns.exception
+import logging
+import threadpool
+
+class CheckZone(object):
+    def __init__(self, zone, nodes, serial, start_period):
+        self.zone          = zone
+        self.master_serial = serial
+        self.nodes         = nodes
+        self.start_period  = start_period
+        self.node_cnt      = len(nodes)
+        self.nodes_report  = {}
+        logging.debug('INIT:{} = {}'.format(self.zone, self.master_serial))
+
+    def _get_serial(self, nameserver):
+        logging.debug('fetching: {}@{}'.format(self.zone, nameserver))
+        request = dns.message.make_query(self.zone, dns.rdatatype.SOA)
+        try:
+            ''' The timeout might need tuning '''
+            response = dns.query.udp(request, nameserver, timeout=.25)
+        except dns.exception.Timeout:
+            logging.error('Timeout from {}@{}'.format(self.zone, nameserver))
+            return False
+        else:
+            return response.answer[0].to_text().split()[6]
+
+    def _check_serial(self, node_dict):
+        ''' TODO: Serial number arithmetic? '''
+        node = node_dict.items()[0][0]
+        addr = node_dict.items()[0][1]
+        node_serial = self._get_serial(addr)
+        porpagation_time = False
+        logging.debug('SERIAL:{}@{} = {}'.format(self.zone, addr, node_serial))
+        if node_serial == str(self.master_serial):
+            porpagation_time = time.time()
+            logging.info('SERIAL:{}@{} == {} @ {}'.format(self.zone, addr, node_serial, 
+                porpagation_time))
+        elif node_serial > str(self.master_serial):
+            logging.debug('SERIAL:{}@{} {} > {} @ {}'.format(self.zone, addr, node_serial, 
+                self.master_serial, time.time() ))
+            porpagation_time = time.time()
+        elif not node_serial:
+            porpagation_time = 'Timeout'
+        else:
+            logging.debug('SERIAL:{}@{} {} != {} @ {}'.format(self.zone, addr, node_serial, 
+                self.master_serial, time.time() ))
+        return (node, porpagation_time)
+
+
+    def _thread_callback(self, request, porpagation_time_tuple):
+        node = porpagation_time_tuple[0]
+        porpagation_time = porpagation_time_tuple[1]
+        if porpagation_time:
+            self.nodes_report[node] = porpagation_time
+
+    def check_propagation(self):
+        ''' Create threadpool before while loop and destroy it after the loop
+            or you get a thread leak! '''
+        main = threadpool.ThreadPool(self.node_cnt)
+        while self.node_cnt != len(self.nodes_report):
+            data = [ dict({node : self.nodes[node]}) for node in self.nodes 
+                    if node not in self.nodes_report ]
+            requests = threadpool.makeRequests(self._check_serial, 
+                    data, self._thread_callback)
+            [main.putRequest(req) for req in requests]
+            main.wait()
+            logging.info('processed: {}/{}'.format(len(self.nodes_report), self.node_cnt))
+            if self.node_cnt != len(self.nodes_report):
+                ''' Report resolution is in seconds so sleep for a bit '''
+                time.sleep(0.25)
+        main.dismissWorkers(self.node_cnt,True)
+        logging.info('{}: processed propagation:'.format(self.zone))
+ 
+def get_enabled_hosts(yaml_dir):
+    nodes = {}
+    for document in glob.glob(os.path.join(yaml_dir, '*.l.root-servers.org.yaml')):
+        node_name = os.path.basename(document).split('.')[0]
+        document = file(document, 'r')
+        yaml_obj = yaml.load(document)
+        logging.debug('YAML:ADD:{}'.format(node_name))
+        if not yaml_obj['status']['operational']:
+            continue
+        for interface, config in yaml_obj['network::interfaces'].items():
+            if 'gw4' in config:
+                nodes[node_name] = config['addr4'].split('/')[0]
+                break
+    return nodes
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser = argparse.ArgumentParser(description='Preform RSSAC propogation measurment')
+    parser.add_argument('--yaml-dir', default='/etc/hierdata/icann-nodes/')
+    parser.add_argument('-v', '--verbose', action='count')
+    parser.add_argument('zone')
+    return parser.parse_args()
+
+def main():
+    args      = parse_args()
+    log_level = logging.ERROR
+    nodes     = get_enabled_hosts(args.yaml_dir)
+
+    if args.verbose == 1:
+        log_level = logging.WARN
+    elif args.verbose == 2:
+        log_level = logging.INFO
+    elif args.verbose == 3:
+        log_level = logging.DEBUG
+    logging.basicConfig(level=log_level)
+    
+
+
+
+if __name__ == "__main__":
+    main()
diff --git a/src/rssac_server/rssac_server.in b/src/rssac_server/rssac_server.in
new file mode 100755
index 0000000..2a929fc
--- /dev/null
+++ b/src/rssac_server/rssac_server.in
@@ -0,0 +1,331 @@
+#!/usr/bin/env python
+
+# 
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+# Original developed by john.bond@icann.org
+# Modified for use with Hedgehog by Sinodun IT (www.sinodun.com)
+
+import sys
+import time
+import rssac
+import socket
+import argparse
+import datetime
+import logging
+import StringIO
+import SocketServer
+import dns.name
+import dns.tsig
+import dns.query
+import dns.message
+import dns.tsigkeyring
+import psycopg2
+import os, pwd, grp
+import yaml
+
+class DnsReaderServer(SocketServer.UDPServer):
+    '''
+    SocketServer.ThreadingUDPServer 
+
+    Instance variables:
+    
+    - RequestHandlerClass
+    '''
+
+    keyname   = None
+    keyring   = None
+    keyalgorithm = None
+    
+    def __init__(self, server_address, RequestHandlerClass, server_name, zone_name, key=None, keyname=None, keyalgo=None):
+        SocketServer.UDPServer.__init__(
+                self, server_address, RequestHandlerClass)
+        
+        config_doc = file('@HEDGEHOGYAML@/hedgehog.yaml', 'r')
+        config = yaml.load(config_doc)
+        config_doc.close()
+        db_host = config['database']['host']
+        db_port = config['database']['port']
+        db_name = config['database']['name']
+        db_owner =  config['database']['owner']
+        db_password = config['database']['owner_pass']
+                
+        ''' We are bound to any restricted ports by this stage '''
+        ''' Now do everything as a non-root user '''
+        ''' Get the uid/gid from the db_owner name '''
+        ''' This assumes that db_owner == username to execute as '''
+        running_uid = pwd.getpwnam(db_owner).pw_uid
+        running_gid = grp.getgrnam(db_owner).gr_gid
+        ''' Clear group privileges '''
+        os.setgroups([])
+        ''' set new uid/gid '''
+        os.setgid(running_gid)
+        os.setuid(running_uid)
+        
+        ''' Start logger '''
+        self.logger = logging.getLogger('rssac_propagation.server.DnsReaderServer')
+        self.zone_name = zone_name
+        
+        if key and keyname:
+            self.keyname          = keyname
+            self.keyring          = dns.tsigkeyring.from_text({ keyname : key})
+        if keyalgo:
+            self.keyalgorithm     = dns.name.from_text(keyalgo)
+            
+        ''' Set up database connection '''
+        self.conn = psycopg2.connect(database=db_name, user=db_owner, password=db_password,
+                                host=db_host, port=db_port)
+        cur = self.conn.cursor()
+        sql = "select version from dsc.version"
+        self.logger.debug('SQL Query: {}'.format(sql))
+        cur.execute(sql)
+        api_tuple = cur.fetchone()
+        if api_tuple == None:
+            cur.close()
+            self.logger.error('SQL Result: Databae API version not defined in the database')
+            raise ValueError('Databae API version not defined in the database')
+        if api_tuple[0] != @DB_VERSION@:
+            cur.close()
+            self.logger.error('SQL Result: Incorrect database API is {} should be {}'.format(api_tuple[0], @DB_VERSION@))
+            raise ValueError('Incorrect database API')
+        self.logger.debug('SQL Result: Database API is {}'.format(api_tuple[0]))
+        
+        ''' Get plot id '''
+        self.load_time_plot_id = self._get_plot_id('load_time')
+        self.zone_size_plot_id = self._get_plot_id('zone_size')
+
+        ''' Get server id '''
+        sql = "SELECT id from server where display_name=%s;"
+        data = (server_name, )
+        fsql = cur.mogrify(sql, data)
+        self.logger.debug('SQL Query: {}'.format(fsql))
+        cur.execute(sql, data)
+        server_id_tuple = cur.fetchone()
+        if server_id_tuple == None:
+            cur.close()
+            raise ValueError('Server is not defined in the database')
+        self.logger.debug('SQL Result: Server {} has id {}'.format(server_name, server_id_tuple[0]))
+        self.db_server_id = server_id_tuple[0]
+        
+        ''' Get the list of nodes to query '''
+        self.nodes = {}
+        sql="SELECT id, ip FROM node WHERE server_id=%s and ip is not null"
+        data = (self.db_server_id, )
+        fsql = cur.mogrify(sql, data)
+        self.logger.debug('SQL: {}'.format(fsql))
+        cur.execute(sql, data)
+        for node in cur:
+            self.nodes[node[0]] = node[1]
+        if not self.nodes:
+            cur.close()
+            raise ValueError('No nodes found for server id {} with an IP address'.format(self.db_server_id))
+        self.logger.debug('SQL Result: Server id {} has IP Adresses for node(s) id {}'.format(self.db_server_id, self.nodes))
+    
+        ''' Get max serial number '''
+        ''' This is read at startup and state is maintained while the server is running '''
+        sql = "SELECT max(data.key2) FROM data WHERE plot_id=%s AND server_id=%s"
+        data = (self.load_time_plot_id, self.db_server_id)
+        fsql = cur.mogrify(sql, data)
+        self.logger.debug('SQL: {}'.format(fsql))
+        cur.execute(sql, data)
+        max_serial_tuple = cur.fetchone()
+        if max_serial_tuple[0] == None:
+            self.max_serial=0
+        else:
+            self.max_serial=max_serial_tuple[0]
+    
+        cur.close()
+        
+    def _get_plot_id(self, plot_name):
+        cur = self.conn.cursor()
+        sql = "SELECT id from dataset where name=%s;"
+        data = (plot_name, )
+        fsql = cur.mogrify(sql, data)
+        self.logger.debug('SQL Query: {}'.format(fsql))
+        cur.execute(sql, data)
+        plot_id_tuple = cur.fetchone()
+        if plot_id_tuple == None:
+            cur.close()
+            raise ValueError('Plot is not defined in the database')
+        self.logger.debug('SQL Result: Plot {} has id {}'.format(plot_name, plot_id_tuple[0]))
+        cur.close()
+        return plot_id_tuple[0]
+        
+    
+        
+class DnsReaderHandler(SocketServer.BaseRequestHandler):
+    '''
+    Base Handler class 
+    '''
+
+    message  = None
+    serial   = None
+    data     = None
+    socket   = None
+    qname    = None
+
+    def __init__(self, request, client_address, server):
+        SocketServer.BaseRequestHandler.__init__(self, request, client_address, server)
+
+    def _process_load_time(self):
+        start = time.time()
+        self.server.logger.debug('SQL Result: Server id {} has max serial {}'.format(self.server.db_server_id, self.server.max_serial))
+        zone_check = rssac.CheckZone(self.qname, self.server.nodes,
+                self.serial, start)
+        zone_check.check_propagation()
+        cur = self.server.conn.cursor()
+        for node, end in zone_check.nodes_report.items():
+            if type(end) is str:
+                ''' Timeout '''
+                load_time = None
+                continue
+            
+            load_time = end - start
+            sql = "INSERT INTO data (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (%s,%s,%s,%s,%s,%s,%s)"
+            data = (datetime.datetime.fromtimestamp(start), self.server.db_server_id, node, self.server.load_time_plot_id, self.qname, self.serial, load_time)
+            fsql = cur.mogrify(sql, data)
+            self.server.logger.debug('SQL: {}'.format(fsql))
+            cur.execute(sql, data)
+        
+        '''' record this serial as the new max(serial) '''
+        self.server.max_serial = self.serial
+        self.server.conn.commit()
+        cur.close()
+        
+    def _get_zone_size(self):
+        self.server.logger.debug('Starting get_zone_size')
+        zone      = StringIO.StringIO()
+        zone_size = 0
+        self.server.logger.debug('xfr query address {} qname {} keyname {}'.format(self.client_address[0], self.qname, self.server.keyname))
+        xfr       = dns.query.xfr(self.client_address[0], self.qname, keyname=self.server.keyname,
+                    keyring=self.server.keyring, keyalgorithm=self.server.keyalgorithm)
+        try:
+            for message in xfr:
+                for ans in message.answer:
+                    ans.to_wire(zone, origin=dns.name.root)
+        except dns.exception.FormError:
+            self.server.logger.error('Error attempting AXFR from {}'.format(self.client_address[0]))
+        else:
+            ''' Need to subtract 2 bytes for the TCP length field '''
+            zone_size = sys.getsizeof(zone.getvalue()) - 2
+        zone.close()
+        self.server.logger.debug('Done get_zone_size')
+        return zone_size
+
+    def _process_zone_size(self):
+        self.server.logger.debug('Starting process_zone_size')
+        start = time.time()
+        zone_size = self._get_zone_size()
+        self.server.logger.debug('zone_size {}'.format(zone_size))
+        if zone_size == 0:
+            ''' Return because we didn't get an answer '''
+            self.server.logger.warn('Warning: No zone size returned from {}'.format(self.client_address[0]))
+            return
+        cur = self.server.conn.cursor()
+        for node in self.server.nodes.keys():
+            sql = "INSERT INTO data (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (%s,%s,%s,%s,%s,%s,%s)"
+            data = (datetime.datetime.fromtimestamp(start), self.server.db_server_id, node, self.server.zone_size_plot_id, self.qname, self.serial, zone_size)
+            fsql = cur.mogrify(sql, data)
+            self.server.logger.debug('SQL: {}'.format(fsql))
+            cur.execute(sql, data)
+            self.server.conn.commit()
+        cur.close()    
+        self.server.logger.debug('Done process_zone_size')
+        
+    def send_response(self):
+        ''' Send notify response '''
+        response = dns.message.make_response(self.message)
+        self.socket.sendto(response.to_wire(), self.client_address)
+        
+    def parse_dns(self):
+        '''
+        parse the data package into dns elements
+        '''
+        self.data = str(self.request[0]).strip()
+        self.socket = self.request[1]
+        #incoming Data
+        try:
+            self.message = dns.message.from_wire(self.data)
+        except dns.name.BadLabelType:
+            #Error processing label (bit flip?)
+            self.server.logger.error('Received Bad label Type from {}'.format(self.client_address[0]))
+        except dns.message.ShortHeader:
+            #Received junk
+            self.server.logger.error('Received Junk from {}'.format(self.client_address[0]))
+        else:
+            current_time = int(time.time())
+            if self.message.opcode() == 4:
+                self.qname = self.message.question[0].name.to_text()
+                if self.qname == self.server.zone_name:
+                    self.server.logger.debug('Received notify for zone {}'.format(self.qname))
+                    if len(self.message.answer) > 0:
+                        answer = self.message.answer[0]
+                        self.serial = answer.to_rdataset()[0].serial
+                    
+                        if int(self.serial) <= int(self.server.max_serial):
+                            self.server.logger.debug('{}:{}:load-time already processed or lower then max({})'.format(
+                                self.qname, self.serial, self.server.max_serial))
+                            self.send_response()
+                            return False
+                        else:    
+                            self.server.logger.debug('Received notify for {} from {}'.format(self.serial, self.client_address[0]))
+                            self.send_response()
+                            return True
+                    else:
+                        self.server.logger.error('Received notify with no serial from {}'.format(self.client_address[0]))
+                        self.send_response()
+                else:
+                    self.server.logger.debug('Received notify for zone {} expected zone {}, ignoring'.format(self.server.zone_name, self.qname))
+        return False
+
+    def handle(self):
+        '''
+        RequestHandlerClass handle function
+        handler listens for dns packets
+        '''
+        if self.parse_dns():
+            self._process_load_time()
+            self._process_zone_size()
+    
+def main():
+    ''' parse cmd line args '''
+    parser = argparse.ArgumentParser(description='nofify receiver')
+    parser.add_argument('-n','--tsig-name')
+    parser.add_argument('-k','--tsig-key')
+    parser.add_argument('-a','--tsig-algo', 
+            choices=['hmac-sha1', 'hmac-sha224', 'hmac-sha256', 'hmac-sha384','hmac-sha512'])
+    parser.add_argument('-v', '--verbose', action='count', help='increase verbosity (up to -vvv)')
+    parser.add_argument('-o','--log', default='server.log')
+    parser.add_argument('-l', '--listen', metavar="0.0.0.0:53", 
+            default="0.0.0.0:53", help='listen on address:port ')
+    parser.add_argument('-s','--server', help='server name', required=True)
+    parser.add_argument('-z','--zone',help='zone name', required=True)
+    args = parser.parse_args()
+    host, port = args.listen.split(":")
+    
+    ''' configure logging '''
+    log_level = logging.ERROR
+    log_format ='%(asctime)s:%(levelname)s:%(name)s:%(funcName)s(%(levelno)s):%(message)s'
+    if args.verbose == 1:
+        log_level = logging.WARN
+    elif args.verbose == 2:
+        log_level = logging.INFO
+    elif args.verbose > 2:
+        log_level = logging.DEBUG
+    logging.basicConfig(level=log_level, filename=args.log, format=log_format)
+    logger = logging.getLogger('rssac_propagation.server')
+    logger.info('rssacd starting up...')
+    
+    ''' Init and then run the server '''
+    server = DnsReaderServer((host, int(port)), DnsReaderHandler, args.server, args.zone,
+            args.tsig_key, args.tsig_name, args.tsig_algo) 
+    server.serve_forever()
+    
+if __name__ == "__main__":
+    main()
\ No newline at end of file
diff --git a/src/rssac_server/rssacd.conf b/src/rssac_server/rssacd.conf
new file mode 100644
index 0000000..d56135c
--- /dev/null
+++ b/src/rssac_server/rssacd.conf
@@ -0,0 +1,10 @@
+TSIG_NAME=""
+TSIG_KEY=""
+# One of hmac-sha1,hmac-sha224,hmac-sha256,hmac-sha384,hmac-sha512
+TSIG_ALGO=""
+LOG="/var/log/hedgehog/rssacd.log"
+LISTEN="0.0.0.0:53"
+# Server name 
+SERVER="Server-A"
+ZONE="."
+#OTHER_ARGS="-v"
diff --git a/src/rssac_server/rssacd.in b/src/rssac_server/rssacd.in
new file mode 100755
index 0000000..dab6c81
--- /dev/null
+++ b/src/rssac_server/rssacd.in
@@ -0,0 +1,17 @@
+#!@PYTHON_EXE@
+
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+#
+# Original developed by john.bond@icann.org
+# Modified for use with Hedgehog by Sinodun IT (www.sinodun.com)
+
+import sys
+sys.path.append('@HHPYSITEDIR@')
+from rssac_server import main
+main()
\ No newline at end of file
diff --git a/src/rssac_server/rssacd.upstart.in b/src/rssac_server/rssacd.upstart.in
new file mode 100755
index 0000000..b3daf68
--- /dev/null
+++ b/src/rssac_server/rssacd.upstart.in
@@ -0,0 +1,11 @@
+description "RSSAC02 Monitor"
+start on filesystem or runlevel [2345]
+stop on runlevel [!2345]
+
+respawn
+respawn limit 10 5
+umask 022
+script 
+ . @HEDGEHOGYAML@/rssacd.conf
+ exec @SBIN@/rssacd -n $TSIG_NAME -k $TSIG_KEY -a $TSIG_ALGO -o $LOG --listen $LISTEN --server $SERVER --zone $ZONE $OTHER_ARGS
+end script
diff --git a/src/rssac_server/threadpool.py b/src/rssac_server/threadpool.py
new file mode 100755
index 0000000..1b4c12c
--- /dev/null
+++ b/src/rssac_server/threadpool.py
@@ -0,0 +1,418 @@
+# -*- coding: UTF-8 -*-
+"""Easy to use object-oriented thread pool framework.
+
+A thread pool is an object that maintains a pool of worker threads to perform
+time consuming operations in parallel. It assigns jobs to the threads
+by putting them in a work request queue, where they are picked up by the
+next available thread. This then performs the requested operation in the
+background and puts the results in another queue.
+
+The thread pool object can then collect the results from all threads from
+this queue as soon as they become available or after all threads have
+finished their work. It's also possible, to define callbacks to handle
+each result as it comes in.
+
+The basic concept and some code was taken from the book "Python in a Nutshell,
+2nd edition" by Alex Martelli, O'Reilly 2006, ISBN 0-596-10046-9, from section
+14.5 "Threaded Program Architecture". I wrapped the main program logic in the
+ThreadPool class, added the WorkRequest class and the callback system and
+tweaked the code here and there. Kudos also to Florent Aide for the exception
+handling mechanism.
+
+Basic usage::
+
+    >>> pool = ThreadPool(poolsize)
+    >>> requests = makeRequests(some_callable, list_of_args, callback)
+    >>> [pool.putRequest(req) for req in requests]
+    >>> pool.wait()
+
+See the end of the module code for a brief, annotated usage example.
+
+Website : http://chrisarndt.de/projects/threadpool/
+
+"""
+__docformat__ = "restructuredtext en"
+
+__all__ = [
+    'makeRequests',
+    'NoResultsPending',
+    'NoWorkersAvailable',
+    'ThreadPool',
+    'WorkRequest',
+    'WorkerThread'
+]
+
+__author__ = "Christopher Arndt"
+__version__ = '1.2.7'
+__revision__ = "$Revision: 416 $"
+__date__ = "$Date: 2009-10-07 05:41:27 +0200 (Wed, 07 Oct 2009) $"
+__license__ = "MIT license"
+
+
+# standard library modules
+import sys
+import threading
+import Queue
+import traceback
+
+
+# exceptions
+class NoResultsPending(Exception):
+    """All work requests have been processed."""
+    pass
+
+class NoWorkersAvailable(Exception):
+    """No worker threads available to process remaining requests."""
+    pass
+
+
+# internal module helper functions
+def _handle_thread_exception(request, exc_info):
+    """Default exception handler callback function.
+
+    This just prints the exception info via ``traceback.print_exception``.
+
+    """
+    traceback.print_exception(*exc_info)
+
+
+# utility functions
+def makeRequests(callable_, args_list, callback=None,
+        exc_callback=_handle_thread_exception):
+    """Create several work requests for same callable with different arguments.
+
+    Convenience function for creating several work requests for the same
+    callable where each invocation of the callable receives different values
+    for its arguments.
+
+    ``args_list`` contains the parameters for each invocation of callable.
+    Each item in ``args_list`` should be either a 2-item tuple of the list of
+    positional arguments and a dictionary of keyword arguments or a single,
+    non-tuple argument.
+
+    See docstring for ``WorkRequest`` for info on ``callback`` and
+    ``exc_callback``.
+
+    """
+    requests = []
+    for item in args_list:
+        if isinstance(item, tuple):
+            requests.append(
+                WorkRequest(callable_, item[0], item[1], callback=callback,
+                    exc_callback=exc_callback)
+            )
+        else:
+            requests.append(
+                WorkRequest(callable_, [item], None, callback=callback,
+                    exc_callback=exc_callback)
+            )
+    return requests
+
+
+# classes
+class WorkerThread(threading.Thread):
+    """Background thread connected to the requests/results queues.
+
+    A worker thread sits in the background and picks up work requests from
+    one queue and puts the results in another until it is dismissed.
+
+    """
+
+    def __init__(self, requests_queue, results_queue, poll_timeout=5, **kwds):
+        """Set up thread in daemonic mode and start it immediatedly.
+
+        ``requests_queue`` and ``results_queue`` are instances of
+        ``Queue.Queue`` passed by the ``ThreadPool`` class when it creates a new
+        worker thread.
+
+        """
+        threading.Thread.__init__(self, **kwds)
+        self.setDaemon(1)
+        self._requests_queue = requests_queue
+        self._results_queue = results_queue
+        self._poll_timeout = poll_timeout
+        self._dismissed = threading.Event()
+        self.start()
+
+    def run(self):
+        """Repeatedly process the job queue until told to exit."""
+        while True:
+            if self._dismissed.isSet():
+                # we are dismissed, break out of loop
+                break
+            # get next work request. If we don't get a new request from the
+            # queue after self._poll_timout seconds, we jump to the start of
+            # the while loop again, to give the thread a chance to exit.
+            try:
+                request = self._requests_queue.get(True, self._poll_timeout)
+            except Queue.Empty:
+                continue
+            else:
+                if self._dismissed.isSet():
+                    # we are dismissed, put back request in queue and exit loop
+                    self._requests_queue.put(request)
+                    break
+                try:
+                    result = request.callable(*request.args, **request.kwds)
+                    self._results_queue.put((request, result))
+                except:
+                    request.exception = True
+                    self._results_queue.put((request, sys.exc_info()))
+
+    def dismiss(self):
+        """Sets a flag to tell the thread to exit when done with current job."""
+        self._dismissed.set()
+
+
+class WorkRequest:
+    """A request to execute a callable for putting in the request queue later.
+
+    See the module function ``makeRequests`` for the common case
+    where you want to build several ``WorkRequest`` objects for the same
+    callable but with different arguments for each call.
+
+    """
+
+    def __init__(self, callable_, args=None, kwds=None, requestID=None,
+            callback=None, exc_callback=_handle_thread_exception):
+        """Create a work request for a callable and attach callbacks.
+
+        A work request consists of the a callable to be executed by a
+        worker thread, a list of positional arguments, a dictionary
+        of keyword arguments.
+
+        A ``callback`` function can be specified, that is called when the
+        results of the request are picked up from the result queue. It must
+        accept two anonymous arguments, the ``WorkRequest`` object and the
+        results of the callable, in that order. If you want to pass additional
+        information to the callback, just stick it on the request object.
+
+        You can also give custom callback for when an exception occurs with
+        the ``exc_callback`` keyword parameter. It should also accept two
+        anonymous arguments, the ``WorkRequest`` and a tuple with the exception
+        details as returned by ``sys.exc_info()``. The default implementation
+        of this callback just prints the exception info via
+        ``traceback.print_exception``. If you want no exception handler
+        callback, just pass in ``None``.
+
+        ``requestID``, if given, must be hashable since it is used by
+        ``ThreadPool`` object to store the results of that work request in a
+        dictionary. It defaults to the return value of ``id(self)``.
+
+        """
+        if requestID is None:
+            self.requestID = id(self)
+        else:
+            try:
+                self.requestID = hash(requestID)
+            except TypeError:
+                raise TypeError("requestID must be hashable.")
+        self.exception = False
+        self.callback = callback
+        self.exc_callback = exc_callback
+        self.callable = callable_
+        self.args = args or []
+        self.kwds = kwds or {}
+
+    def __str__(self):
+        return "<WorkRequest id=%s args=%r kwargs=%r exception=%s>" % \
+            (self.requestID, self.args, self.kwds, self.exception)
+
+class ThreadPool:
+    """A thread pool, distributing work requests and collecting results.
+
+    See the module docstring for more information.
+
+    """
+
+    def __init__(self, num_workers, q_size=0, resq_size=0, poll_timeout=5):
+        """Set up the thread pool and start num_workers worker threads.
+
+        ``num_workers`` is the number of worker threads to start initially.
+
+        If ``q_size > 0`` the size of the work *request queue* is limited and
+        the thread pool blocks when the queue is full and it tries to put
+        more work requests in it (see ``putRequest`` method), unless you also
+        use a positive ``timeout`` value for ``putRequest``.
+
+        If ``resq_size > 0`` the size of the *results queue* is limited and the
+        worker threads will block when the queue is full and they try to put
+        new results in it.
+
+        .. warning:
+            If you set both ``q_size`` and ``resq_size`` to ``!= 0`` there is
+            the possibilty of a deadlock, when the results queue is not pulled
+            regularly and too many jobs are put in the work requests queue.
+            To prevent this, always set ``timeout > 0`` when calling
+            ``ThreadPool.putRequest()`` and catch ``Queue.Full`` exceptions.
+
+        """
+        self._requests_queue = Queue.Queue(q_size)
+        self._results_queue = Queue.Queue(resq_size)
+        self.workers = []
+        self.dismissedWorkers = []
+        self.workRequests = {}
+        self.createWorkers(num_workers, poll_timeout)
+
+    def createWorkers(self, num_workers, poll_timeout=5):
+        """Add num_workers worker threads to the pool.
+
+        ``poll_timout`` sets the interval in seconds (int or float) for how
+        ofte threads should check whether they are dismissed, while waiting for
+        requests.
+
+        """
+        for i in range(num_workers):
+            self.workers.append(WorkerThread(self._requests_queue,
+                self._results_queue, poll_timeout=poll_timeout))
+
+    def dismissWorkers(self, num_workers, do_join=False):
+        """Tell num_workers worker threads to quit after their current task."""
+        dismiss_list = []
+        for i in range(min(num_workers, len(self.workers))):
+            worker = self.workers.pop()
+            worker.dismiss()
+            dismiss_list.append(worker)
+
+        if do_join:
+            for worker in dismiss_list:
+                worker.join()
+        else:
+            self.dismissedWorkers.extend(dismiss_list)
+
+    def joinAllDismissedWorkers(self):
+        """Perform Thread.join() on all worker threads that have been dismissed.
+        """
+        for worker in self.dismissedWorkers:
+            worker.join()
+        self.dismissedWorkers = []
+
+    def putRequest(self, request, block=True, timeout=None):
+        """Put work request into work queue and save its id for later."""
+        assert isinstance(request, WorkRequest)
+        # don't reuse old work requests
+        assert not getattr(request, 'exception', None)
+        self._requests_queue.put(request, block, timeout)
+        self.workRequests[request.requestID] = request
+
+    def poll(self, block=False):
+        """Process any new results in the queue."""
+        while True:
+            # still results pending?
+            if not self.workRequests:
+                raise NoResultsPending
+            # are there still workers to process remaining requests?
+            elif block and not self.workers:
+                raise NoWorkersAvailable
+            try:
+                # get back next results
+                request, result = self._results_queue.get(block=block)
+                # has an exception occured?
+                if request.exception and request.exc_callback:
+                    request.exc_callback(request, result)
+                # hand results to callback, if any
+                if request.callback and not \
+                       (request.exception and request.exc_callback):
+                    request.callback(request, result)
+                del self.workRequests[request.requestID]
+            except Queue.Empty:
+                break
+
+    def wait(self):
+        """Wait for results, blocking until all have arrived."""
+        while 1:
+            try:
+                self.poll(True)
+            except NoResultsPending:
+                break
+
+
+################
+# USAGE EXAMPLE
+################
+
+if __name__ == '__main__':
+    import random
+    import time
+
+    # the work the threads will have to do (rather trivial in our example)
+    def do_something(data):
+        time.sleep(random.randint(1,5))
+        result = round(random.random() * data, 5)
+        # just to show off, we throw an exception once in a while
+        if result > 5:
+            raise RuntimeError("Something extraordinary happened!")
+        return result
+
+    # this will be called each time a result is available
+    def print_result(request, result):
+        print "**** Result from request #%s: %r" % (request.requestID, result)
+
+    # this will be called when an exception occurs within a thread
+    # this example exception handler does little more than the default handler
+    def handle_exception(request, exc_info):
+        if not isinstance(exc_info, tuple):
+            # Something is seriously wrong...
+            print request
+            print exc_info
+            raise SystemExit
+        print "**** Exception occured in request #%s: %s" % \
+          (request.requestID, exc_info)
+
+    # assemble the arguments for each job to a list...
+    data = [random.randint(1,10) for i in range(20)]
+    # ... and build a WorkRequest object for each item in data
+    requests = makeRequests(do_something, data, print_result, handle_exception)
+    # to use the default exception handler, uncomment next line and comment out
+    # the preceding one.
+    #requests = makeRequests(do_something, data, print_result)
+
+    # or the other form of args_lists accepted by makeRequests: ((,), {})
+    data = [((random.randint(1,10),), {}) for i in range(20)]
+    requests.extend(
+        makeRequests(do_something, data, print_result, handle_exception)
+        #makeRequests(do_something, data, print_result)
+        # to use the default exception handler, uncomment next line and comment
+        # out the preceding one.
+    )
+
+    # we create a pool of 3 worker threads
+    print "Creating thread pool with 3 worker threads."
+    main = ThreadPool(3)
+
+    # then we put the work requests in the queue...
+    for req in requests:
+        main.putRequest(req)
+        print "Work request #%s added." % req.requestID
+    # or shorter:
+    # [main.putRequest(req) for req in requests]
+
+    # ...and wait for the results to arrive in the result queue
+    # by using ThreadPool.wait(). This would block until results for
+    # all work requests have arrived:
+    # main.wait()
+
+    # instead we can poll for results while doing something else:
+    i = 0
+    while True:
+        try:
+            time.sleep(0.5)
+            main.poll()
+            print "Main thread working...",
+            print "(active worker threads: %i)" % (threading.activeCount()-1, )
+            if i == 10:
+                print "**** Adding 3 more worker threads..."
+                main.createWorkers(3)
+            if i == 20:
+                print "**** Dismissing 2 worker threads..."
+                main.dismissWorkers(2)
+            i += 1
+        except KeyboardInterrupt:
+            print "**** Interrupted!"
+            break
+        except NoResultsPending:
+            print "**** No pending results."
+            break
+    if main.dismissedWorkers:
+        print "Joining all dismissed worker threads..."
+        main.joinAllDismissedWorkers()
diff --git a/tools/Makefile.am b/tools/Makefile.am
new file mode 100755
index 0000000..67d8878
--- /dev/null
+++ b/tools/Makefile.am
@@ -0,0 +1,190 @@
+CLEANFILES = $(bin_SCRIPTS)
+
+bin_SCRIPTS = refile_and_grok \
+              hedgehog/hedgehog_database_create \
+              hedgehogctl
+
+nobase_libexec_SCRIPTS = hedgehog/conf_read \
+                         hedgehog/check_user_perms \
+                         hedgehog/database_init \
+                         hedgehog/database_manage_partitions \
+                         hedgehog/database_update_geoip \
+                         hedgehog/database_update_nodes \
+                         hedgehog/database_update_service_addr \
+                         hedgehog/database_update_tlds \
+                         hedgehog/database_update_tlds_from_zone \
+                         hedgehog/datafiles_create_summary \
+                         hedgehog/datafiles_rm_empty_xml_dirs \
+                         hedgehog/datafiles_tar_old_xml_files \
+                         hedgehog/plotcache_generate_cached_plots \
+                         hedgehog/plotcache_rm_cached_plots \
+                         hedgehog/rssac_generate_reports \
+                         hedgehog/database_process_rssac_data \
+                         hedgehog/database_rm_old_rssac_data \
+                         hedgehog/DDL_updates/000010_ddl_python \
+                         hedgehog/DDL_updates/000011_ddl_new_graphs \
+                         hedgehog/DDL_updates/000012_ddl_lower_key_index \
+                         hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn \
+                         hedgehog/DDL_updates/000014_ddl_service_addr \
+                         hedgehog/DDL_updates/000015_ddl_byregion \
+                         hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans \
+                         hedgehog/DDL_updates/000017_ddl_node_stats \
+                         hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX \
+                         hedgehog/utility_prepare_GeoLite2-City-Locations-en
+
+nobase_data_DATA = $(srcdir)/hedgehog/sql/initialise_database.sql \
+                   $(srcdir)/hedgehog/sql/insert_lookup_data.sql \
+                   $(srcdir)/hedgehog/sql/dns.sql
+
+subst_LIBEXECDATA = sed -e "s|@LIBEXEC@|@libexecdir@/hedgehog|" -e "s|@DATA@|@datadir@|"
+subst_DB_VERSION = sed -e "s|@DB_VERSION@|@DB_SCHEMA_VERSION@|"
+subst_BIN = sed -e "s|@BIN@|@bindir@|"
+do_subst_python = sed -e 's|@PYTHON_EXE@|$(PYTHON)|' -e 's|@HHPYSITEDIR@|$(pythondir)|'
+subst_sql = sed -e 's|<SQL_PATH>|$(datadir)/hedgehog/sql|'
+subst_version = sed -e 's|<VERSION>|@VERSION@|'
+
+hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX: $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in Makefile
+	$(do_subst_python) < $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in > hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX
+
+hedgehog/utility_prepare_GeoLite2-City-Locations-en: $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in Makefile
+	$(do_subst_python) < $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in > hedgehog/utility_prepare_GeoLite2-City-Locations-en
+	
+refile_and_grok: refile_and_grok.in Makefile
+	$(subst_LIBEXECDATA) < $(srcdir)/refile_and_grok.in | $(subst_BIN) > refile_and_grok
+
+hedgehogctl: hedgehogctl.in Makefile
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehogctl.in > hedgehogctl
+
+hedgehog/conf_read: $(srcdir)/hedgehog/conf_read.in Makefile
+	mkdir -p hedgehog
+	$(subst_HH_CONF_FILE) < $(srcdir)/hedgehog/conf_read.in > hedgehog/conf_read
+
+hedgehog/check_user_perms: $(srcdir)/hedgehog/check_user_perms.in Makefile
+	mkdir -p hedgehog
+	$(subst_DB_VERSION) < $(srcdir)/hedgehog/check_user_perms.in | $(subst_LIBEXECDATA) | $(subst_version) > hedgehog/check_user_perms
+	
+hedgehog/database_init: $(srcdir)/hedgehog/database_init.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_init.in > hedgehog/database_init
+
+hedgehog/database_manage_partitions: $(srcdir)/hedgehog/database_manage_partitions.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_manage_partitions.in > hedgehog/database_manage_partitions
+
+hedgehog/database_update_geoip: $(srcdir)/hedgehog/database_update_geoip.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_geoip.in > hedgehog/database_update_geoip
+
+hedgehog/database_update_nodes: $(srcdir)/hedgehog/database_update_nodes.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_nodes.in > hedgehog/database_update_nodes
+
+hedgehog/database_update_service_addr: $(srcdir)/hedgehog/database_update_service_addr.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_service_addr.in > hedgehog/database_update_service_addr
+
+hedgehog/database_update_tlds: $(srcdir)/hedgehog/database_update_tlds.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_tlds.in > hedgehog/database_update_tlds
+
+hedgehog/database_update_tlds_from_zone: $(srcdir)/hedgehog/database_update_tlds_from_zone.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_tlds_from_zone.in > hedgehog/database_update_tlds_from_zone
+
+hedgehog/datafiles_create_summary: $(srcdir)/hedgehog/datafiles_create_summary.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/datafiles_create_summary.in > hedgehog/datafiles_create_summary
+
+hedgehog/datafiles_rm_empty_xml_dirs: $(srcdir)/hedgehog/datafiles_rm_empty_xml_dirs.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/datafiles_rm_empty_xml_dirs.in > hedgehog/datafiles_rm_empty_xml_dirs
+
+hedgehog/datafiles_tar_old_xml_files: $(srcdir)/hedgehog/datafiles_tar_old_xml_files.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/datafiles_tar_old_xml_files.in > hedgehog/datafiles_tar_old_xml_files
+
+hedgehog/plotcache_generate_cached_plots: $(srcdir)/hedgehog/plotcache_generate_cached_plots.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/plotcache_generate_cached_plots.in > hedgehog/plotcache_generate_cached_plots
+
+hedgehog/plotcache_rm_cached_plots: $(srcdir)/hedgehog/plotcache_rm_cached_plots.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/plotcache_rm_cached_plots.in > hedgehog/plotcache_rm_cached_plots
+
+hedgehog/rssac_generate_reports: $(srcdir)/hedgehog/rssac_generate_reports.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/rssac_generate_reports.in > hedgehog/rssac_generate_reports
+
+hedgehog/database_process_rssac_data: $(srcdir)/hedgehog/database_process_rssac_data.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_process_rssac_data.in > hedgehog/database_process_rssac_data
+
+hedgehog/database_rm_old_rssac_data: $(srcdir)/hedgehog/database_rm_old_rssac_data.in Makefile
+	mkdir -p hedgehog
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_rm_old_rssac_data.in > hedgehog/database_rm_old_rssac_data
+
+hedgehog/DDL_updates/000010_ddl_python: $(srcdir)/hedgehog/DDL_updates/000010_ddl_python.in Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000010_ddl_python.in | $(subst_DB_VERSION) | $(subst_sql) > hedgehog/DDL_updates/000010_ddl_python
+
+hedgehog/DDL_updates/000011_ddl_new_graphs: $(srcdir)/hedgehog/DDL_updates/000011_ddl_new_graphs.in Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000011_ddl_new_graphs.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000011_ddl_new_graphs
+
+hedgehog/DDL_updates/000012_ddl_lower_key_index: $(srcdir)/hedgehog/DDL_updates/000012_ddl_lower_key_index.in Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000012_ddl_lower_key_index.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000012_ddl_lower_key_index
+
+hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn: $(srcdir)/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in  | $(subst_DB_VERSION) > hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn
+
+hedgehog/DDL_updates/000014_ddl_service_addr: $(srcdir)/hedgehog/DDL_updates/000014_ddl_service_addr.in  Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000014_ddl_service_addr.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000014_ddl_service_addr
+
+hedgehog/DDL_updates/000015_ddl_byregion: $(srcdir)/hedgehog/DDL_updates/000015_ddl_byregion.in Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000015_ddl_byregion.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000015_ddl_byregion
+
+hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans: $(srcdir)/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans
+
+hedgehog/DDL_updates/000017_ddl_node_stats: $(srcdir)/hedgehog/DDL_updates/000017_ddl_node_stats.in Makefile
+	mkdir -p hedgehog/DDL_updates
+	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000017_ddl_node_stats.in | $(subst_DB_VERSION) >hedgehog/DDL_updates/000017_ddl_node_stats
+
+EXTRA_DIST = refile_and_grok.in \
+             conf_read.in \
+             database_init.in \
+             database_manage_partitions.in \
+             database_update_nodes.in \
+             datafiles_create_summary.in \
+             datafiles_rm_empty_xml_dirs.in \
+             datafiles_tar_old_xml_files.in \
+             plotcache_generate_cached_plots.in \
+             plotcache_rm_cached_plots.in \
+             rssac_generate_reports.in \
+             database_process_rssac_data.in \
+             database_rm_old_rssac_data.in 
+
+install-data-hook:
+	mkdir -p ${DESTDIR}$(localstatedir)/hedgehog/data
+	chmod -R 775 ${DESTDIR}$(localstatedir)/hedgehog/data
+
+
+dist_man_MANS = hedgehogctl.1 \
+                refile_and_grok.1 \
+                hedgehog_database_create.1
+
+hedgehogctl.1: hedgehogctl Makefile
+	chmod 755 hedgehogctl
+	$(HELP2MAN) -h help2man --version-string=@PACKAGE_VERSION@ -N -s 1 -n "Wrapper around hedgehog management scripts" ./hedgehogctl -o hedgehogctl.1
+	
+refile_and_grok.1: refile_and_grok Makefile
+	chmod 755 refile_and_grok
+	$(HELP2MAN) -h -h --version-string=@PACKAGE_VERSION@ -N -s 1 -n "Wrapper around the dsc-extractor application that launches multiple extractors to process data" ./refile_and_grok -o refile_and_grok.1
+
+hedgehog_database_create.1: $(srcdir)/hedgehog/hedgehog_database_create Makefile
+	$(HELP2MAN) -h -h --version-string=@PACKAGE_VERSION@ -N -s 1 -n "This script is to aid the database administrator with creating the hedgehog database." $(srcdir)/hedgehog/hedgehog_database_create -o hedgehog_database_create.1
\ No newline at end of file
diff --git a/tools/hedgehog/DDL_updates/000010_ddl_python.in b/tools/hedgehog/DDL_updates/000010_ddl_python.in
new file mode 100755
index 0000000..0ec4b3b
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000010_ddl_python.in
@@ -0,0 +1,88 @@
+#!/bin/bash
+#
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000010_ddl_python
+#
+
+eval $(@LIBEXEC@/conf_read)
+
+DB_NAME=$hhconfig_database_name
+DATA_DIR=$hhconfig_directories_data
+SQ="'"
+
+NEW_DDL_SERIAL=10
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=8
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))
+
+DSC_SCHEMA="dsc"
+
+usage () {
+  echo
+  echo "Usage: $(basename $0) options"
+  echo
+  echo "Supported options:"
+  echo "  -d Run in debug mode"
+  echo "  -h Show this help."
+  echo "  -P <password for the postgresql super user>"
+  echo "  -U <username for the postgresql super user> Required"
+  echo "  -v Show version"
+  exit 1
+}
+
+while getopts ":hP:U:v" opt; do
+  case $opt in
+    d  ) set -x ;;
+    h  ) usage ;;
+    P  ) SPECIFIED_PASS=$OPTARG ;;
+    U  ) SPECIFIED_USER=$OPTARG ;;
+    v  ) echo "<VERSION>"
+         exit 0 ;;
+    \? ) usage ;;
+  esac
+done
+
+if [ -z "$SPECIFIED_USER" ] ; then
+  usage
+fi
+
+# We need a superuser. 
+a=$(@LIBEXEC@/check_user_perms -U $SPECIFIED_USER -P $SPECIFIED_PASS -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
+if [ $? -ne 0 ] ; then
+  echo $a
+  exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+echo "$(date): $0 COMMENCED"
+
+psql $CONN_STRING -qtc '\dL' | grep plpythonu
+if [ $? -eq 1 ] ; then
+  echo "No plpythonu language installed. Ensure that the plpythonu package is installed."
+  echo "Then connect to the $DB_NAME database as a superuser and run the command \"CREATE EXTENSION plpythonu SCHEMA pg_catalog;\""
+  exit 1
+fi
+
+psql $CONN_STRING < <SQL_PATH>/dns.sql
+if [ $? -eq 0 ] ; then
+psql $CONN_STRING <<EOF
+BEGIN;
+
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new ip2asn functions', now() );
+COMMIT;
+EOF
+fi
+
+echo "`date`: $0 COMPLETED"
diff --git a/tools/hedgehog/DDL_updates/000011_ddl_new_graphs.in b/tools/hedgehog/DDL_updates/000011_ddl_new_graphs.in
new file mode 100755
index 0000000..2ffd982
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000011_ddl_new_graphs.in
@@ -0,0 +1,271 @@
+#!/bin/bash
+#
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000011_ddl_new_graphs
+#
+
+eval $(@LIBEXEC@/conf_read)
+
+DB_NAME=$hhconfig_database_name
+DATA_DIR=$hhconfig_directories_data
+SQ="'"
+export PGPASSWORD=$hhconfig_database_owner_pass
+
+NEW_DDL_SERIAL=11
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=9
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))
+
+DSC_SCHEMA="dsc"
+
+usage () {
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+    exit 1
+}
+
+
+while getopts ":csh" opt; do
+    case $opt in
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+# Need a write user to update DB
+echo "`date`: $0 COMMENCED"
+a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
+if [ $? -ne 0 ] ; then
+  echo $a
+  exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+echo "Starting"
+
+# START TLD
+TLD_DATASET=$(psql -h $hhconfig_database_host $DB_NAME -qtc "SELECT id FROM dataset WHERE name = 'qtype_vs_tld';")
+# END  TLD
+# START GEOIP
+psql -h $hhconfig_database_host $DB_NAME -qtc '\dx' | grep ip4r | grep dsc
+[ $? -eq 1 ] && \
+echo "No ip4r extension installed. Ensure that the ip4r package is installed." && \
+echo "Then connect to the $DB_NAME database as a superuser and run the command \"CREATE EXTENSION ip4r SCHEMA dsc;\"" && \
+exit 1
+GEO_DATASET=$(psql -h $hhconfig_database_host $DB_NAME -qtc "SELECT id FROM dataset WHERE name = 'client_subnet_accum';")
+# END GEOIP
+# START IP2ASN
+IP2ASN_DATASET=$(psql $CONN_STRING -qtc "SELECT id FROM dataset WHERE name = 'client_subnet_accum';")
+IP2ASN_DATASET2=$(psql $CONN_STRING -qtc "SELECT id FROM dataset WHERE name = 'client_addr_vs_rcode_accum';")
+IP2ASN_DATASET3=$(psql $CONN_STRING -qtc "SELECT id FROM dataset WHERE name = 'client_subnet2_accum';")
+# END IP2ASN
+
+echo "Adding graphs"
+
+
+psql -h $hhconfig_database_host $DB_NAME <<EOF
+BEGIN;
+-- START TLDs
+CREATE TABLE tld_types (id serial PRIMARY KEY, type_name text);
+INSERT INTO tld_types (type_name) VALUES
+  ('New-gTLD'),
+  ('Legacy-gTLD'),
+  ('ccTLD');
+CREATE TABLE tlds (id serial PRIMARY KEY,
+                   alabel text UNIQUE,
+                   type int REFERENCES tld_types (id));
+
+-- INSERT INTO dataset (name) VALUES ('# client_subnet_vs_tld');
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('qtype_vs_cctld','3.1.QTYPE vs TLD','QTYPE for most popular ccTLDs','QTYPE value for most popular ccTLDs queried') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${TLD_DATASET} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title)
+    VALUES ('qtype_vs_newgtld','3.1.QTYPE vs TLD','QTYPE for most popular New gTLDs','QTYPE value for most popular New gTLDs queried') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${TLD_DATASET} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('qtype_vs_legacygtld','3.1.QTYPE vs TLD','QTYPE for most popular Legacy gTLDs','QTYPE value for most popular Legacy gTLDs queried') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot select id, ${TLD_DATASET} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('qtype_vs_othertld','3.1.QTYPE vs TLD','QTYPE for most popular Undelegated TLDs','QTYPE value for most popular Undelegated TLDs queried') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${TLD_DATASET} FROM rows;
+
+/* WITH rows AS (
+*   INSERT INTO visible_plot (name, display_category, display_name, title)
+*     VALUES ('client_subnet_vs_tld','7.Client Subnet Statistics','Busiest client subnet vs TLD','Client subnet vs TLD')
+*     RETURNING id
+* )
+* INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='client_subnet_vs_tld') FROM rows;
+*/
+
+UPDATE visible_plot set display_category = '3.1.QTYPE vs TLD' where name='qtype_vs_tld';
+UPDATE visible_plot set display_name='RCODE by client subnet (fixed size)' where name='client_addr_vs_rcode_accum';
+-- END TLDs
+
+-- START GEO
+DROP TABLE IF EXISTS geo;
+DROP SEQUENCE IF EXISTS geo_id_seq;
+
+CREATE TABLE geoip (
+  ip_range iprange PRIMARY KEY,
+  latitude real,
+  longitude real,
+  geoname integer );
+
+CREATE INDEX ix_ip_range ON geoip USING gist (ip_range);
+
+CREATE TABLE locations (
+  geoname_id integer PRIMARY KEY,
+  name text,
+  city text );
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('geomap','10.GEO','Busiest client locations by city','Busiest client locations by city') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${GEO_DATASET} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('geochart','10.GEO','Busiest client locations by country','Busiest client locations by country, node locations by city') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${GEO_DATASET} FROM rows;
+-- END GEO
+
+-- START NODE GROUPING
+ALTER TABLE node  ADD column subgroup character varying(255);
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('by_instance','1.Node Statistics','By instance','Queries grouped by node instance') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='qtype') FROM rows;
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('by_city','1.Node Statistics','By city','Queries grouped by city') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='qtype') FROM rows;
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('by_country','1.Node Statistics','By country','Queries grouped by country') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='qtype') FROM rows;
+-- END NODE GROUPING
+
+-- START RSSAC
+INSERT INTO dataset (name) VALUES ('load_time');
+INSERT INTO dataset (name) VALUES ('zone_size');
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('load_time','9.RSSAC','Zone propagation time','Latency in the distribution system by serial number') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='load_time') FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('zone_size','9.RSSAC','Zone size','Size of the zone by serial number') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='zone_size') FROM rows;
+
+ALTER table node add ip ipaddress;
+-- END RSSAC
+
+-- STRART IP2ANS
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('asn','7.Client Subnet Statistics','Busiest client ASNs','Client ASNs for the busiest /24(IPv4) or /96(IPv6)') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('bgpprefix','7.Client Subnet Statistics','Busiest client BGP prefix','Clients by BGP prefix for the busiest /24(IPv4) or /96(IPv6)') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('client_addr_vs_rcode_accum_asn','7.Client Subnet Statistics', 'RCODE by client ASNs', 'RCODE by client ASNs for the busiest /24(IPv4) or /96(IPv6)') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET2} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('client_addr_vs_rcode_accum_bgpprefix','7.Client Subnet Statistics', 'RCODE by clients by BGP prefix', 'RCODE by client BGP prefix for the busiest /24(IPv4) or /96(IPv6)')
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET2} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('client_subnet2_accum_asn','8.Classification', 'Query classification by client ASNs', 'Query classification by client ASNs for the busiest /24(IPv4) or /96(IPv6)') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET3} FROM rows;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('client_subnet2_accum_bgpprefix','8.Classification', 'Query classification by clients by BGP prefix', 'Query classification by client BGP prefix for the busiest /24(IPv4) or /96(IPv6)')
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET3} FROM rows;
+
+UPDATE visible_plot SET title='Busiest client subnets (fixed IPv4/8 or IPv6/32)', display_name='Busiest client subnets (fixed size)' where name='client_subnet_accum';
+-- END IP2ASN
+
+-- START DEST ADDR
+INSERT INTO dataset (name) VALUES ('server_addr');
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('server_addr','11.Server IP','By server IP address','Server IP address') 
+    RETURNING id
+) 
+INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='server_addr') FROM rows;
+-- END DEST ADDR
+
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
+COMMIT;
+EOF
+
+echo "`date`: $0 COMPLETED"
diff --git a/tools/hedgehog/DDL_updates/000012_ddl_lower_key_index.in b/tools/hedgehog/DDL_updates/000012_ddl_lower_key_index.in
new file mode 100755
index 0000000..dd47710
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000012_ddl_lower_key_index.in
@@ -0,0 +1,108 @@
+#!/bin/bash
+#
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000009_ddl_lower_key_index
+#
+
+eval $(@LIBEXEC@/conf_read)
+
+DB_NAME=$hhconfig_database_name
+DATA_DIR=$hhconfig_directories_data
+SQ="'"
+PREDICT=0;
+SUPPRESS_WARNINGS=0
+export PGPASSWORD=$hhconfig_database_owner_pass
+
+set -e
+NEW_DDL_SERIAL=12
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=9
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 0 ))
+
+DSC_SCHEMA="dsc"
+
+CONCURRENT=0
+SKIP=0
+
+usage () {
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -c Build the indexes with the CONCURRENTLY flag set. This is slower"
+    echo "     but allows writes to continue being made to the tables."
+    echo "  -s Skip creating the new indexes. The new indexes should improve"
+    echo "     performance but are optional. You still need to run this script to"
+    echo "     update version numbers correctly in the database."
+    echo "  -h Show this help."
+    exit 1
+}
+
+
+while getopts ":csh" opt; do
+    case $opt in
+        c  ) CONCURRENT=1 ;;
+        s  ) SKIP=1 ;;
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+echo "`date`: $0 COMMENCED"
+
+if [ $SKIP -eq 0 ] ; then
+  echo "This script creates new indexes on all the data tables. It will take a long"
+  echo "time to complete if you have a lot of data. In addition it requires a lock"
+  echo "against writes to the tables being updated (reads are fine) if not run"
+  echo "concurrently."
+  echo "You should either:"
+  echo "1. Stop database writes while this script is running"
+  echo "2. Set the -c flag to build the indexes concurrently"
+  echo "3. Set the -s flag to skip index creation."
+  echo
+  echo "See the usage (-h) for more details."
+  echo
+  echo "Do you wish to continue? [yN]"
+  read c
+  [[ ! "$c" =~ ^(Y|y) ]] && exit 1
+fi
+
+USER=$(whoami)
+[ $USER != $hhconfig_database_owner ] && echo "Must be $hhconfig_database_owner to run this script" && exit 1
+
+DB_API_VERSION=`psql -h $hhconfig_database_host $DB_NAME -tc  "select version from dsc.version;"`
+[ $DB_API_VERSION != $REQUIRED_API_VERSION ] && echo "Error: Database API version incorrect." && exit 1
+
+LAST_SERIAL=`psql -h $hhconfig_database_host $DB_NAME -tc  "select max(serial) from dsc.internal_version;"`
+[ $LAST_SERIAL != $REQUIRED_DDL_SERIAL ] && echo "Error: ddl scripts not run in correct order." && exit 1
+
+# Create node_id indexes on all data tables
+if [ $SKIP -eq 0 ] ; then
+  MYFILE=$(mktemp)
+  if [ $CONCURRENT -eq 0 ] ; then
+    psql -h $hhconfig_database_host $DB_NAME -tc "select 'create index idx_' || table_name || '_lower_key1 on ' || table_name || ' (lower(key1));' FROM information_schema.tables where table_name like 'data\_%' and table_type='BASE TABLE' and table_schema='dsc';" > $MYFILE
+  else
+    psql -h $hhconfig_database_host $DB_NAME -tc "select 'create index CONCURRENTLY idx_' || table_name || '_lower_key1 on ' || table_name || ' (lower(key1));' FROM information_schema.tables where table_name like 'data\_%' and table_type='BASE TABLE' and table_schema='dsc';" > $MYFILE
+  fi
+  psql -h $hhconfig_database_host $DB_NAME < $MYFILE
+  rm $MYFILE
+fi
+
+psql -h $hhconfig_database_host $DB_NAME <<EOF
+BEGIN;
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added lower case index to data tables', now() );
+COMMIT;
+EOF
+
+echo "`date`: $0 COMPLETED"
diff --git a/tools/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in b/tools/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in
new file mode 100755
index 0000000..cc01049
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in
@@ -0,0 +1,83 @@
+#!/bin/bash
+#
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000013_ddl_dnskey_vs_asn
+#
+
+eval $(@LIBEXEC@/conf_read)
+
+DB_NAME=$hhconfig_database_name
+DATA_DIR=$hhconfig_directories_data
+export PGPASSWORD=$hhconfig_database_owner_pass
+
+NEW_DDL_SERIAL=13
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=10
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))
+
+DSC_SCHEMA="dsc"
+
+usage () {
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+    exit 1
+}
+
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+# Need a write user to update DB
+echo "`date`: $0 COMMENCED"
+a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
+if [ $? -ne 0 ] ; then
+  echo $a
+  exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+echo "Starting"
+psql -h $hhconfig_database_host $DB_NAME <<EOF
+BEGIN;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('dnskey_vs_asn','7.Client Subnet Statistics','DNSKEY Queries by ASN','DNSKEY Queries by ASN') 
+    RETURNING id
+), rows2 AS (
+  INSERT INTO dataset (name) VALUES ('qtype_vs_client_subnet')
+    RETURNING id as dataset_id
+)
+INSERT INTO dataset_visible_plot SELECT id, dataset_id FROM rows, rows2;
+
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
+COMMIT;
+EOF
+
+echo "`date`: $0 COMPLETED"
+
+
+
+
+
+
+
diff --git a/tools/hedgehog/DDL_updates/000014_ddl_service_addr.in b/tools/hedgehog/DDL_updates/000014_ddl_service_addr.in
new file mode 100755
index 0000000..0de339f
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000014_ddl_service_addr.in
@@ -0,0 +1,61 @@
+#!/bin/bash
+#
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000014_ddl_service_addr
+#
+
+
+NEW_DDL_SERIAL=14
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=11
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))
+
+DSC_SCHEMA="dsc"
+
+usage () {
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+    exit 1
+}
+
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+# Need a write user to update DB
+echo "`date`: $0 COMMENCED"
+a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
+if [ $? -ne 0 ] ; then
+  echo $a
+  exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+echo "Starting"
+psql $CONN_STRING <<EOF
+BEGIN;
+CREATE TABLE $DSC_SCHEMA.service_addr (id SERIAL PRIMARY KEY, server_id INTEGER REFERENCES $DSC_SCHEMA.server (id), address IPADDRESS NOT NULL, UNIQUE (server_id, address));
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added service address table', now() );
+COMMIT;
+EOF
+
+echo "`date`: $0 COMPLETED"
diff --git a/tools/hedgehog/DDL_updates/000015_ddl_byregion.in b/tools/hedgehog/DDL_updates/000015_ddl_byregion.in
new file mode 100755
index 0000000..cdb10c8
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000015_ddl_byregion.in
@@ -0,0 +1,68 @@
+#!/bin/bash
+#
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000015_ddl_byregion
+#
+
+
+NEW_DDL_SERIAL=15
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=12
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))
+
+DSC_SCHEMA="dsc"
+
+usage () {
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+    exit 1
+}
+
+
+while getopts ":csh" opt; do
+    case $opt in
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+# Need a write user to update DB
+echo "`date`: $0 COMMENCED"
+a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
+if [ $? -ne 0 ] ; then
+  echo $a
+  exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+echo "Starting"
+
+psql $CONN_STRING <<EOF
+BEGIN;
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('by_region','1.Node Statistics','By region','Queries by region') 
+    RETURNING id
+)
+INSERT INTO dataset_visible_plot SELECT id, (SELECT id from dataset WHERE name='qtype') FROM rows;
+
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new by region graph', now() );
+COMMIT;
+EOF
+
+echo "`date`: $0 COMPLETED"
\ No newline at end of file
diff --git a/tools/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in b/tools/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in
new file mode 100755
index 0000000..d6c4a13
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in
@@ -0,0 +1,83 @@
+#!/bin/bash
+#
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000016_ddl_server_addr_vs_trans
+#
+
+eval $(@LIBEXEC@/conf_read)
+
+DB_NAME=$hhconfig_database_name
+DATA_DIR=$hhconfig_directories_data
+export PGPASSWORD=$hhconfig_database_owner_pass
+
+NEW_DDL_SERIAL=16
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=13
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))
+
+DSC_SCHEMA="dsc"
+
+usage () {
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+    exit 1
+}
+
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+# Need a write user to update DB
+echo "`date`: $0 COMMENCED"
+a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
+if [ $? -ne 0 ] ; then
+  echo $a
+  exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+echo "Starting"
+psql -h $hhconfig_database_host $DB_NAME <<EOF
+BEGIN;
+
+WITH rows AS (
+  INSERT INTO visible_plot (name, display_category, display_name, title) 
+    VALUES ('server_addr_vs_trans','11.Server IP','By server IP address and transport','Server IP address by transport') 
+    RETURNING id
+), rows2 AS (
+  INSERT INTO dataset (name) VALUES ('server_addr_vs_trans')
+    RETURNING id as dataset_id
+)
+INSERT INTO dataset_visible_plot SELECT id, dataset_id FROM rows, rows2;
+
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
+COMMIT;
+EOF
+
+echo "`date`: $0 COMPLETED"
+
+
+
+
+
+
+
diff --git a/tools/hedgehog/DDL_updates/000017_ddl_node_stats.in b/tools/hedgehog/DDL_updates/000017_ddl_node_stats.in
new file mode 100755
index 0000000..057c8eb
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/000017_ddl_node_stats.in
@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   000017_ddl_node_stats
+#
+
+
+NEW_DDL_SERIAL=17
+REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
+NEW_API_VERSION=14
+REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))
+
+DSC_SCHEMA="dsc"
+
+usage () {
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+    exit 1
+}
+
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+# Need a write user to update DB
+echo "`date`: $0 COMMENCED"
+a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
+if [ $? -ne 0 ] ; then
+  echo $a
+  exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+echo "Starting"
+psql $CONN_STRING <<EOF
+BEGIN;
+UPDATE visible_plot set display_category='1.Query Statistics' 
+  WHERE display_category='1.Node Statistics';
+  
+UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
+INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
+COMMIT;
+EOF
+
+echo "`date`: $0 COMPLETED"
\ No newline at end of file
diff --git a/tools/hedgehog/DDL_updates/README.txt b/tools/hedgehog/DDL_updates/README.txt
new file mode 100644
index 0000000..b24a906
--- /dev/null
+++ b/tools/hedgehog/DDL_updates/README.txt
@@ -0,0 +1,14 @@
+This directory contains scripts that are used to update the database
+between releases. The database keeps a record of which of these has
+been run. They should be run in numerical order.
+
+2.0.0 -> 2.1.0
+---------------
+To upgrade from 2.0.0 to 2.1.0b1 run the following scripts in this order
+
+* as super user of hedgehog database (e.g. postgres)
+  000010_ddl_python
+
+* as the DB write user (e.g. hedgehog):
+  000011_ddl_new_graphs
+  000012_ddl_lower_key_index
diff --git a/tools/hedgehog/check_user_perms.in b/tools/hedgehog/check_user_perms.in
new file mode 100755
index 0000000..d269449
--- /dev/null
+++ b/tools/hedgehog/check_user_perms.in
@@ -0,0 +1,214 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   check_user_perms
+#
+
+# This script should be called like this:
+# a=$(@LIBEXEC@/check_user_perms -wfc)
+# if [ $? -ne 0 ] ; then
+#   echo $a
+#   exit 1
+# fi
+# eval $a
+# export PGPASSWORD=$PGPASSWORD
+
+
+eval $(@LIBEXEC@/conf_read)
+
+usage () {
+  echo
+  echo "Check if we are a valid read user and check that we can connect to the DB"
+  echo "Optionally specify DB version checking, directory permission checking"
+  echo "or check for write access to the DB."
+  echo
+  echo "Usage: $(basename "$0") options"
+  echo
+  echo "Supported options:"
+  echo "  -C <API_VERSION> specify DB API version to check for (implies -c)"
+  echo "  -n Indicate that the user doesn't need to access the database"
+  echo "  -s <SERIAL_NUMBER> check DB serial number"
+  echo "  -c check databse API version against the build time API version"
+  echo "  -f check user has write permission to $hhconfig_directories_data"
+  echo "  -g check user has write permission to web directory ${hhconfig_directories_rssac%rssac}"
+  echo "  -p check user has write permission to plotcache directory $hhconfig_directories_plotcache"
+  echo "  -r check user has write permission to rssac directory ${hhconfig_directories_rssac}"
+  echo "  -w check we are a write user of the DB"
+  echo "  -U specify a DB user"
+  echo "  -P specify a DB password"
+  echo "  -d run in debug mode"
+  echo "  -h Show this help"
+}
+
+# Read only access by default
+WRITER=0
+CHECK_API=0
+CHECK_SERIAL=0
+CHECK_DATA_DIR=0
+CHECK_WEB_DIR=0
+CHECK_RSSAC_DIR=0
+CHECK_PLOT_CACHE_DIR=0
+SPECIFIED_PASS=""
+SPECIFIED_USER=""
+# By default check for a database connection
+CHECK_DB_CONN=1
+MY_USER=$(whoami)
+
+REQUIRED_API_VERSION=@DB_VERSION@
+
+while getopts ":C:ncdfgprP:s:U:whv" opt; do
+    case $opt in
+    C  ) REQUIRED_API_VERSION=$OPTARG
+         CHECK_API=1 ;;
+    n  ) CHECK_DB_CONN=0 ;;
+    c  ) CHECK_API=1 ;;
+    d  ) set -x ;;
+    f  ) CHECK_DATA_DIR=1 ;;
+    g  ) CHECK_WEB_DIR=1 ;;
+    p  ) CHECK_PLOT_CACHE_DIR=1 ;;
+    P  ) SPECIFIED_PASS=$OPTARG ;;
+    p  ) CHECK_RSSAC_DIR=1 ;;
+    s  ) CHECK_SERIAL=1
+         REQUIRED_DDL_SERIAL=$OPTARG ;;
+    U  ) SPECIFIED_USER=$OPTARG ;;
+    w  ) WRITER=1 ;;
+    h  ) usage
+         exit 0 ;;
+    v  ) echo "<VERSION>"
+         exit 0 ;;
+    \? ) echo "Unknown arguments. Use -h for help"
+         exit 1 ;;
+    esac
+done
+if [ $CHECK_DB_CONN -eq 1 ] ; then
+
+  USER_NAME=$hhconfig_database_read_user
+  if [ -z "$USER_NAME" ]; then
+    echo "Database user name not set"
+    exit 1
+  fi
+  export PGPASSWORD="$hhconfig_database_read_pass"
+  if [ $WRITER -eq 1 ] ; then
+    USER_NAME=$hhconfig_database_owner
+    export PGPASSWORD="$hhconfig_database_owner_pass"
+  fi
+  if [ -n "$SPECIFIED_USER" ] ; then
+    USER_NAME=$SPECIFIED_USER
+    export PGPASSWORD="$SPECIFIED_PASS"
+  fi
+
+  # If the password is set then assume md5 authentication otherwise you must be
+  # a user with the same name as the hedgehog database owner and are using peer
+  # authentication
+  if [ -z "$PGPASSWORD" ] ; then
+    if [ "$MY_USER" != "$USER_NAME" ] ; then
+      echo "Must be user: '$USER_NAME' and using peer authentication to run this script, currently '$MY_USER'"
+      exit 1
+    fi
+  fi
+
+  PORT=$hhconfig_database_port
+  if [ -z "$PORT" ] ; then
+    PORT=5432
+  fi
+  DB_NAME=$hhconfig_database_name
+  if [ -z "$DB_NAME" ] ; then
+    echo "Database name not set"
+    exit 1
+  fi
+  DB_HOST=$hhconfig_database_host
+  if [ -z "$DB_HOST" ]; then
+    echo "Database host not set"
+    exit 1
+  fi
+
+  CONN_STRING="-h $DB_HOST -U $USER_NAME -p $PORT -d $DB_NAME"
+  CONN_STRING_XX="host=$DB_HOST port=$PORT dbname=$DB_NAME password=$PGPASSWORD"
+  # Test connection to the database
+  psql $CONN_STRING -tc "\conninfo" > /dev/null
+  if [ $? -ne 0 ] ; then
+    echo "Error connecting to database using connection string $CONN_STRING"
+    exit 1
+  fi
+
+  # Now test the optional parameters...
+  TEST=$(psql $CONN_STRING -tc "SELECT has_table_privilege('dsc.version', 'SELECT')" | tr -d '[:space:]')
+  if [ $WRITER -eq 1 ] ; then
+    TEST=$(psql $CONN_STRING -tc "SELECT has_table_privilege('dsc.version', 'INSERT')" | tr -d '[:space:]')
+  fi
+
+  if [ "x$TEST" == "xt" ] ; then
+    if [ $CHECK_API -eq 1 ] ; then
+      DB_API_VERSION=$(psql $CONN_STRING -tc  "select version from dsc.version;")
+      if [ $DB_API_VERSION -ne $REQUIRED_API_VERSION ] ; then
+        echo "Error: Database API version incorrect: Found $DB_API_VERSION expected $REQUIRED_API_VERSION"
+        exit 1
+      fi
+    fi
+  else
+    echo "The user '$USER_NAME' does not have the expected database access."
+    exit 1
+  fi
+fi
+
+if [ $CHECK_SERIAL -eq 1 ] ; then
+  LAST_SERIAL=$(psql $CONN_STRING -tc "select max(serial) from dsc.internal_version;")
+  if [ $LAST_SERIAL -ne $REQUIRED_DDL_SERIAL ] ; then
+    echo "Error: DDL scripts have not been run in correct order. Required $REQUIRED_DDL_SERIAL, actual version is $LAST_SERIAL"
+    exit 1
+  fi
+fi
+
+if [ $CHECK_DATA_DIR -eq 1 ] ; then
+  OWNER=$(stat -c %U $hhconfig_directories_data)
+  if [ "$OWNER" != "$MY_USER" ] ; then
+    echo "The user '$MY_USER'  does not have write access to the data dirctory $hhconfig_directories_data owned by '$OWNER'"
+    exit 1
+  fi
+fi
+
+if [ $CHECK_WEB_DIR -eq 1 ] ; then
+  # The %rssac removes rssac from the end of the path, thus checking the www directory instead.
+  mydir=${hhconfig_directories_rssac/%rssac/}
+  WWW_OWNER=$(stat -c %U $mydir)
+  if [ "$WWW_OWNER" != "$MY_USER" ] ; then
+    echo "The user '$MY_USER'  does not have write access to the www dirctory $mydir owned by '$WWW_OWNER'"
+    exit 1
+  fi
+fi
+
+if [ $CHECK_PLOT_CACHE_DIR -eq 1 ] ; then
+  OWNER=$(stat -c %U $hhconfig_directories_plotcache)
+  if [ "$OWNER" != "$MY_USER" ] ; then
+    echo "The user '$MY_USER'  does not have write access to the plotcache dirctory $hhconfig_directories_plotcache owned by '$OWNER'"
+    exit 1
+  fi
+fi
+
+if [ $CHECK_RSSAC_DIR -eq 1 ] ; then
+  OWNER=$(stat -c %U $hhconfig_directories_rssac)
+  if [ "$OWNER" != "$MY_USER" ] ; then
+    echo "The user '$MY_USER'  does not have write access to the rssac dirctory $hhconfig_directories_rssac, owned by '$OWNER'"
+    exit 1
+  fi
+fi
+
+# print env variables for eval to process in calling script
+echo "RSSAC_DIR=\"$hhconfig_directories_rssac\""
+echo "CONF_DIR=\"$hhconfig_directories_conf\""
+echo "PLOT_CACHE_DIR=\"$hhconfig_directories_plotcache\""
+echo "DATADIR=\"$hhconfig_directories_data\""
+echo "PGPASSWORD=\"$PGPASSWORD\""
+echo "CONN_STRING=\"$CONN_STRING\""
+echo "CONN_STRING_XX=\"$CONN_STRING_XX\""
+exit 0
\ No newline at end of file
diff --git a/tools/hedgehog/conf_read.in b/tools/hedgehog/conf_read.in
new file mode 100755
index 0000000..e738166
--- /dev/null
+++ b/tools/hedgehog/conf_read.in
@@ -0,0 +1,25 @@
+#!/usr/bin/perl -w
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   conf_read
+#
+
+use YAML::Tiny;
+$yaml = YAML::Tiny->read("@HH_CONF_FILE@");
+foreach $key (keys  $yaml->[0]{database}){
+  next if (!$yaml->[0]{database}{$key}); 
+  print "hhconfig_database_$key=\"$yaml->[0]{database}{$key}\"\n";
+}
+foreach $key (keys  $yaml->[0]{directories}){
+  print "hhconfig_directories_$key=\"$yaml->[0]{directories}{$key}\"\n";
+}
\ No newline at end of file
diff --git a/tools/hedgehog/database_init.in b/tools/hedgehog/database_init.in
new file mode 100755
index 0000000..08418f8
--- /dev/null
+++ b/tools/hedgehog/database_init.in
@@ -0,0 +1,163 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_init
+#
+
+eval $(@LIBEXEC@/conf_read)
+
+# DSTART defaults to the current month
+DSTART=$(date --date="$(date +%Y-%m-01)" +%Y-%m)
+SQ="'"
+
+usage () {
+	echo
+	echo "Create and populate the database tables with plot options and nodes."
+	echo
+	echo "This calls the hedeghog_database_update_nodes script which will populate the DB with the"
+	echo "specified nodes and create a directory structure for them in the working data directory."
+	echo
+	echo "Usage: $(basename $0) options"
+	echo
+	echo "Supported options:"
+	echo "  -m Start month of oldest data to be imported (default is current month, format YYYY-MM)"
+	echo "  -h Show this help"
+}
+
+while getopts ":u:m:ph" opt; do
+    case $opt in
+        m  ) DSTART=$OPTARG ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+# Validate the input date. Add day since (oddly) date won't parse just year and month
+if [[ ! $DSTART =~ ^[0-9]{4}-[0-1][0-9]$ ]] ; then
+		echo "Error: The start date is not in the required format of YYYY-MM" 
+		exit 1
+fi
+date "+%Y-%m-%d" --date="${DSTART}-01" >/dev/null 2>&1
+is_valid=$?
+if [ ${is_valid} -ne 0 ] ; then
+	echo "Error: The start date is invalid" 
+	exit 1
+fi
+
+echo "Running with the following options:"
+echo "  Host of database....................... $hhconfig_database_host"
+echo "  Port of database....................... $hhconfig_database_port"
+echo "  Name of database....................... $hhconfig_database_name"
+echo "  Owner of database...................... $hhconfig_database_owner"
+echo "  Reader of database..................... $hhconfig_database_read_user"
+echo "  Start date of oldest data.............. $DSTART"
+echo
+  echo  "*** This script will read nodes.csv file in order to  *** "
+  echo  "*** populate the database with servers and nodes.     *** "
+  echo  "*** Ensure this file is up to date before proceeding. *** "
+  echo
+  echo -n "Are you sure you want to continue? y/N: "
+  read c
+  if [ "$c" != "y" ] && [ "$c" != "Y" ]
+  then
+    echo "Script halted by user - script exiting."
+    exit 1
+  fi
+
+export PGPASSWORD=$hhconfig_database_owner_pass
+
+psql -h $hhconfig_database_host -d $hhconfig_database_name <<EOF
+  REVOKE ALL ON DATABASE "$hhconfig_database_name" FROM public;
+  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT SELECT ON TABLES TO hh_read;
+  GRANT CONNECT ON DATABASE "$hhconfig_database_name" TO hh_read;
+  GRANT USAGE ON SCHEMA dsc to hh_read;
+  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT SELECT ON TABLES TO hh_write;
+  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT INSERT ON TABLES TO hh_write;
+  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT UPDATE ON TABLES TO hh_write;
+  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT DELETE ON TABLES TO hh_write;
+  GRANT CONNECT ON DATABASE $hhconfig_database_name TO hh_write;
+  GRANT USAGE ON SCHEMA dsc to hh_write;
+EOF
+
+#################  CREATE TABLES  #########################
+###########################################################
+# Create the base tables, views, functions and constraints
+echo "Initialising the database (creating base tables, views, functions and constraints)"
+psql -h $hhconfig_database_host -d $hhconfig_database_name < @DATA@/hedgehog/sql/initialise_database.sql
+echo
+
+# You have to grant privileges on sequences after creating them
+echo "Granting privileges on sequences to hh_write"
+psql -h $hhconfig_database_host -d $hhconfig_database_name <<EOF
+GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA dsc TO hh_write;
+\q
+EOF
+echo
+###########################################################
+
+###################  POPULATE TABLES  #####################
+###########################################################
+# Populate the dsc.plot and dsc.iana_lookup tables
+echo "Populating lookup tables (dsc.plot, dsc.iana_lookup)"
+psql -h $hhconfig_database_host -d $hhconfig_database_name  < @DATA@/hedgehog/sql/insert_lookup_data.sql #> /dev/null
+###########################################################
+
+###################  RUN DDL scripts  #####################
+###########################################################
+# Run any scripts needed to update the DB
+echo "Running DDL scripts"
+# 000010_ddl not run, as requires super user. Done in DB create.
+@LIBEXEC@/DDL_updates/000011_ddl_new_graphs
+# DB schema now contains the new index, therefore only run script to update version
+@LIBEXEC@/DDL_updates/000012_ddl_lower_key_index -s
+# Add a dataset/plot  for dnskey_vs_asn
+@LIBEXEC@/DDL_updates/000013_ddl_dnskey_vs_asn
+# Add a service address table
+@LIBEXEC@/DDL_updates/000014_ddl_service_addr
+# Add by region
+@LIBEXEC@/DDL_updates/000015_ddl_byregion
+# Add server_addr_vs_transport
+@LIBEXEC@/DDL_updates/000016_ddl_server_addr_vs_trans
+# Remove node from web interface
+@LIBEXEC@/DDL_updates/000017_ddl_node_stats
+###########################################################
+
+###################  POPULATE TABLES  #####################
+###########################################################
+echo "Populate servers and nodes"
+@LIBEXEC@/database_update_tlds
+@LIBEXEC@/database_update_geoip
+@LIBEXEC@/database_update_nodes
+@LIBEXEC@/database_update_service_addr
+###########################################################
+
+#############  PARTITION DATA TABLES  #####################
+###########################################################
+# Partition the dsc.data table: create 1 table per month per server per plot
+echo "Creating partitioned tables for dsc.data (1 table per month per server per plot type)"
+echo "This may take some time if there is a lot of historic data to import"
+i=1
+DNOW_FIRST=`date +%Y%m01`
+DNOW=`date +%Y-%m`
+DSTART=`date --date="${DSTART}-01" +%Y%m%d`
+while [ $DNOW_FIRST -ge $DSTART ]
+do
+  DNEXT=`date --date="$DNOW_FIRST + 1 month" +%Y-%m`
+  @LIBEXEC@/database_manage_partitions -m ${DNOW}
+  DNOW_FIRST=`date --date="$DNOW_FIRST - 1 month" +%Y%m%d`
+  DNOW=`date --date="$DNOW_FIRST" +%Y-%m`
+done
+###########################################################
+echo "database_init complete"
diff --git a/tools/hedgehog/database_manage_partitions.in b/tools/hedgehog/database_manage_partitions.in
new file mode 100755
index 0000000..1c5181a
--- /dev/null
+++ b/tools/hedgehog/database_manage_partitions.in
@@ -0,0 +1,106 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_manage_partitions
+
+# We need a write user
+a=$(@LIBEXEC@/check_user_perms -wc)
+if [ $? -ne 0 ] ; then
+	echo "$a"
+	exit 1
+fi
+eval "$a"
+export PGPASSWORD=$PGPASSWORD
+
+usage () {
+	echo
+	echo "Create a new set of partitioned data tables in the database to hold imported data."
+	echo "The partitions are created per month of data."
+	echo
+	echo "Usage: $(basename "$0") options"
+	echo
+	echo "Supported options:"
+	echo " -m Month to create partitions for (default is next month, format YYYY-MM)."
+	echo " -h Show this help."
+}
+
+while getopts ":m:h" opt; do
+    case $opt in
+        m  ) PDATE=$OPTARG ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+if [ -z "$PDATE" ] ; then
+    NOW=$(date --date="+1 month" +%Y_%m)
+    NEXT=$(date --date="+2 month" +%Y_%m)
+else
+	if [[ ! $PDATE =~ ^[0-9]{4}-[0-1][0-9]$ ]] ; then
+			echo "Error: The start date is not in the required format of YYYY-MM" 
+			exit 1
+	fi
+	# Validate the input date. Add day since (oddly) date won't parse just year and month
+	date "+%Y-%m-%d" --date="${PDATE}-01" >/dev/null 2>&1
+	is_valid=$?
+	if [ ${is_valid} -ne 0 ] ; then
+		echo "Error: The start date is invalid" 
+		exit 1
+	fi
+	
+    NOW=$(date --date="$PDATE-01" +%Y_%m)
+    
+    IFS='_' read -a YM <<< "$NOW"
+    YEAR=${YM[0]}
+    MONTH=${YM[1]}
+    NEXT="${YEAR}_$(( ${MONTH#0} + 1))"
+
+    if [ ${MONTH#0} -eq 12 ] ; then
+        NEXT=$((YEAR + 1))"_01"
+    fi
+fi
+
+SERVERS=$(psql $CONN_STRING -tc "SELECT name FROM dsc.server;")
+PLOTS=$(psql $CONN_STRING -tc "SELECT name FROM dsc.dataset;")
+
+for server in $SERVERS
+do
+    s_id=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server WHERE name = '$server';")
+    for plot in $PLOTS
+    do
+      p_id=$(psql $CONN_STRING -tc "SELECT id FROM dsc.dataset WHERE name = '$plot';")
+      
+      psql $CONN_STRING <<EOF
+BEGIN;
+
+CREATE TABLE dsc.data_${server}_${plot}_${NOW} (
+  CHECK (starttime >= TIMESTAMP '$NOW-01 00:00:00' AT TIME ZONE 'UTC' AND starttime < TIMESTAMP '$NEXT-01 00:00:00' AT TIME ZONE 'UTC'),
+  CHECK (plot_id = $p_id),
+  CHECK (server_id = $s_id)
+) INHERITS (dsc.data);
+
+ALTER TABLE dsc.data_${server}_${plot}_${NOW}
+        ADD CONSTRAINT pk_data_${server}_${plot}_${NOW} PRIMARY KEY (server_id, node_id , plot_id , starttime , key1, key2 ),
+        ADD CONSTRAINT fk_data_${server}_${plot}_${NOW}_node FOREIGN KEY (node_id) REFERENCES dsc.node (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION,
+        ADD CONSTRAINT fk_data_${server}_${plot}_${NOW}_plot FOREIGN KEY (plot_id) REFERENCES dsc.dataset (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION;
+CREATE INDEX idx_data_${server}_${plot}_${NOW}_node_id ON dsc.data_${server}_${plot}_${NOW} (node_id);
+COMMIT;
+EOF
+    done
+done
+
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/database_process_rssac_data.in b/tools/hedgehog/database_process_rssac_data.in
new file mode 100755
index 0000000..35db20b
--- /dev/null
+++ b/tools/hedgehog/database_process_rssac_data.in
@@ -0,0 +1,163 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_process_rssac_data
+#
+
+# We need a write user
+a=$(@LIBEXEC@/check_user_perms -wc)
+if [ $? -ne 0 ] ; then
+	echo "$a"
+	exit 1
+fi
+eval "$a"
+export PGPASSWORD=$PGPASSWORD
+
+DEFAULT_DAY=$(date --date="- 7 day" +%Y-%m-%d)
+START=${DEFAULT_DAY}
+FORCE_SUM_DEL=0
+DELETE_RAW=0
+
+usage () {
+    echo
+    echo "Process the raw RSSAC unique source data into a summary table for a specified 24 hour period"
+    echo "(Must be run prior to generating the RSSAC reports for this day with the rssac_generate_reports script)"
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -d Date on which to process data (default is 7 days ago: \"${DEFAULT_DAY}\", format: YYYY-MM-DD)"
+    echo "  -f Force overwrite of existing summary data"
+    echo "  -D Also delete the raw unique_source data from the database to save space (default: no delete)"
+    echo "  -h Show this help"
+}
+
+while getopts ":d:fDh" opt; do
+    case $opt in
+        d  ) START=$OPTARG ;;
+        f  ) FORCE_SUM_DEL=1 ;;
+        D  ) DELETE_RAW=1 ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+
+# Validate the start date
+if [[ ! $START =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]$ ]] ; then
+		echo "Error: The start date is not in the required format of YYYY-MM-DD" 
+		exit 1
+fi
+date "+%Y-%m-%d" --date="${START}" >/dev/null 2>&1
+is_valid=$?
+if [ ${is_valid} -ne 0 ] ; then
+	echo "Error: The start date is invalid" 
+	exit 1
+fi
+
+echo "Processing unique_source data for ${START}"
+
+# Confirm force flag as this should be exceptional
+if [ ${FORCE_SUM_DEL} -eq 1 ] ; then
+	 echo
+	 echo "You have specified the -f flag to force an overwrite of existing summary data"
+	 echo -n "Are you sure you want to continue? y/N: "
+	 read c
+	 if [ "$c" != "y" ] && [ "$c" != "Y" ]
+	 then
+	   echo "Script halted by user - script exiting."
+	   exit 1
+	 fi
+fi
+
+SQ="'"
+SERVER_IDS=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server;")
+UNIQUE_SOURCE_RAW_PLOT_NAME="unique_sources_raw"
+UNIQUE_SOURCE_SUM_PLOT_NAME="unique_sources"
+UNIQUE_SOURCE_RAW_PLOT_ID=$(psql $CONN_STRING -tc "SELECT plot_id FROM dsc.plot where name=${SQ}${UNIQUE_SOURCE_RAW_PLOT_NAME}${SQ};")
+UNIQUE_SOURCE_SUM_PLOT_ID=$(psql $CONN_STRING -tc "SELECT plot_id FROM dsc.plot where name=${SQ}${UNIQUE_SOURCE_SUM_PLOT_NAME}${SQ};")
+
+DELETE_SUMMARY_SQL=""
+DELETE_RAW_SQL=""
+IPV4_key="IPv4"
+IPV6_key="IPv6"
+IPV6_ag_key="IPv6/64"
+IPV4_value=0
+IPV6_value=0
+IPV6_ag_value=0
+
+for SERVER_ID in $SERVER_IDS ;do
+
+	# Set up the table name
+    SERVER_NAME=$(psql $CONN_STRING -tc "SELECT name FROM dsc.server WHERE id=${SERVER_ID};" | tr -d [[:space:]])
+    NODE_COUNT=$(psql $CONN_STRING -tc "SELECT count(*) FROM dsc.node where server_id=${SERVER_ID};")
+    NODE_IDS=$(psql $CONN_STRING -tc "SELECT id FROM dsc.node where server_id=${SERVER_ID} order by id;")
+    MONTH=$(date --date="${START}" +%m)
+    YEAR=$(date --date="${START}" +%Y)
+    # Could do this work directly on dsc.data instead
+    UNIQUE_SOURCE_RAW_TABLE_NAME="dsc.data_${SERVER_NAME}_${UNIQUE_SOURCE_RAW_PLOT_NAME}_${YEAR}_${MONTH}"
+    UNIQUE_SOURCE_SUM_TABLE_NAME="dsc.data_${SERVER_NAME}_${UNIQUE_SOURCE_SUM_PLOT_NAME}_${YEAR}_${MONTH}"
+    echo "Found $NODE_COUNT nodes for server $SERVER_NAME" 
+
+
+   for NODE_ID in $NODE_IDS ; do
+	
+		echo "Processing node id $NODE_ID"
+		# First up, check that there isn't an entry in the summary table for this node. 
+		# Note that if there is actually data when the transaction occurs then the insert will fail and the transaction will rollback.
+		SUMMARY_TABLE_ENTRY=$(psql $CONN_STRING -tc "SELECT count(*) FROM ${UNIQUE_SOURCE_SUM_TABLE_NAME} WHERE node_id=${NODE_ID} and starttime = TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC' ;")
+		if [ ${SUMMARY_TABLE_ENTRY} -ne 0 ] ; then
+			if [ ${FORCE_SUM_DEL} -eq 1 ] ; then
+				echo "Summary data already exists for node id = ${NODE_ID}... Force flag was used overwriting summary data!"
+				DELETE_SUMMARY_SQL="DELETE FROM ${UNIQUE_SOURCE_SUM_TABLE_NAME} WHERE node_id=${NODE_ID} and starttime = TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC' ;"
+			else
+				echo "Summary data already exists for node id = ${NODE_ID}... skipping"
+				continue;
+			fi
+		fi
+		
+		# Now lets get the summary numbers from the raw table if they exist. Reset the counters to 0 first.
+		IPV4_value=0
+		IPV6_value=0
+		IPV6_ag_value=0
+		IPv4_SUMMARY_RESULT=$(psql $CONN_STRING -tc "SELECT y from unique_source_summary_function(${SERVER_ID}, ${UNIQUE_SOURCE_RAW_PLOT_ID}, TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', TIMESTAMP '${START} 23:59:00' AT TIME ZONE 'UTC', '${NODE_ID}') WHERE x='${IPV4_key}';")
+		IPv6_SUMMARY_RESULT=$(psql $CONN_STRING -tc "SELECT y from unique_source_summary_function(${SERVER_ID}, ${UNIQUE_SOURCE_RAW_PLOT_ID}, TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', TIMESTAMP '${START} 23:59:00' AT TIME ZONE 'UTC', '${NODE_ID}') WHERE x='${IPV6_key}';")
+		IPv6_ag_SUMMARY_RESULT=$(psql $CONN_STRING -tc "SELECT y from unique_source_summary_function(${SERVER_ID}, ${UNIQUE_SOURCE_RAW_PLOT_ID}, TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', TIMESTAMP '${START} 23:59:00' AT TIME ZONE 'UTC', '${NODE_ID}') WHERE x='${IPV6_ag_key}';")
+		if [ ! -z "${IPv4_SUMMARY_RESULT}" ] ; then IPV4_value=${IPv4_SUMMARY_RESULT} ; fi
+		if [ ! -z "${IPv6_SUMMARY_RESULT}" ] ; then IPV6_value=${IPv6_SUMMARY_RESULT} ; fi
+		if [ ! -z "${IPv6_ag_SUMMARY_RESULT}" ] ; then IPV6_ag_value=${IPv6_ag_SUMMARY_RESULT} ; fi
+								
+		# Time to delete the raw data if that is what we were told to do						
+		if [ ${DELETE_RAW} -eq 1 ] ; then
+			echo "Deleting raw data for for node id = ${NODE_ID}"
+			DELETE_RAW_SQL="DELETE FROM ${UNIQUE_SOURCE_RAW_TABLE_NAME} WHERE node_id=${NODE_ID} and starttime = TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC' ;"
+		fi
+													
+		psql $CONN_STRING <<EOF
+BEGIN;
+set client_min_messages to WARNING;
+${DELETE_SUMMARY_SQL}
+INSERT INTO ${UNIQUE_SOURCE_SUM_TABLE_NAME} (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', ${SERVER_ID}, ${NODE_ID}, ${UNIQUE_SOURCE_SUM_PLOT_ID}, ${SQ}${IPV4_key}${SQ}, 1, ${IPV4_value});
+INSERT INTO ${UNIQUE_SOURCE_SUM_TABLE_NAME} (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', ${SERVER_ID}, ${NODE_ID}, ${UNIQUE_SOURCE_SUM_PLOT_ID}, ${SQ}${IPV6_key}${SQ}, 1, ${IPV6_value});
+INSERT INTO ${UNIQUE_SOURCE_SUM_TABLE_NAME} (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', ${SERVER_ID}, ${NODE_ID}, ${UNIQUE_SOURCE_SUM_PLOT_ID}, ${SQ}${IPV6_ag_key}${SQ}, 1, ${IPV6_ag_value});
+${DELETE_RAW_SQL}
+COMMIT;
+EOF
+	done
+done
+echo "$(date): $0 COMPLETED"
+exit 0
diff --git a/tools/hedgehog/database_rm_old_rssac_data.in b/tools/hedgehog/database_rm_old_rssac_data.in
new file mode 100755
index 0000000..a912b44
--- /dev/null
+++ b/tools/hedgehog/database_rm_old_rssac_data.in
@@ -0,0 +1,106 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_rm_old_rssac_data
+#
+
+#TODO: Add option to select all servers or just one server
+
+# We need a write user
+a=$(@LIBEXEC@/check_user_perms -wc)
+if [ $? -ne 0 ] ; then
+	echo "$a"
+	exit 1
+fi
+eval "$a"
+export PGPASSWORD=$PGPASSWORD
+
+DEFAULT_DAY=$(date --date="$(date +%Y-%m-15)" +%Y%m%d)
+NOW=$(date --date="now" +%Y%m%d)
+
+if [ $DEFAULT_DAY -ge $NOW ] ; then
+	START=$(date --date="$NOW - 2 month" +%Y-%m)
+else
+	START=$(date --date="$NOW - 1 month" +%Y-%m)
+fi
+
+
+usage () {
+    echo
+    echo "Delete all the raw RSSAC unique_source data in the database for a given month. All data older"
+    echo "than this is also deleted."
+    echo "This script is intended to be run monthly to free disk space on systems where the raw"
+    echo "RSSAC unique sources data becomes unmanagably large."
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -m Month from which to delete all raw unique source data (default ${START}: format YYYY-MM)"
+    echo "       (default is 2 months ago when in the first half of the current month and"
+    echo "        1 month ago when in the second half of the current month)"
+    echo "  -h Show this help"
+}
+
+while getopts ":m:h" opt; do
+    case $opt in
+        m  ) START=$OPTARG ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+# Validate the input date. Add day since (oddly) date won't parse just year and month
+if [ ! -z "$START" ] ; then
+	if [[ ! $START =~ ^[0-9]{4}-[0-1][0-9]$ ]] ; then
+			echo "Error: The start date is not in the required format of YYYY-MM" 
+			exit 1
+	fi
+	date "+%Y-%m-%d" --date="${START}-01" >/dev/null 2>&1
+	is_valid=$?
+	if [ ${is_valid} -ne 0 ] ; then
+		echo "Error: The start date is not valid" 
+		exit 1
+	fi
+	
+	START_FIRST=$(date --date=$START-01 +%Y%m%d)
+	NOW_FIRST=$(date --date="$NOW" +%Y%m01)
+	if [ $NOW_FIRST -le $START_FIRST ] ; then
+		echo "Error: The given month cannot be this month or in the future." 
+		exit 1
+	fi
+fi
+
+echo "Deleting unique_source data during and before ${START}" 
+
+UNIQUE_SOURCE_RAW_PLOT_NAME="unique_sources_raw"
+
+TABLE_NAMES=$(psql $CONN_STRING -tc "select tablename from pg_tables where schemaname='dsc' order by tablename" | grep $UNIQUE_SOURCE_RAW_PLOT_NAME)
+
+START_TRUNC=$(date --date="$START-01" +%Y%m%d)
+for TABLENAME in $TABLE_NAMES
+do
+	(( SIZE=${#TABLENAME}-7 ))
+	TMP_DATE=$(echo ${TABLENAME:$SIZE} | sed -e 's/_/-/')
+	TABLE_DATE=$(date --date="$TMP_DATE-01" +%Y%m%d)
+	if [ $START_TRUNC -ge $TABLE_DATE ] ; then
+		echo "Truncating table $TABLENAME"
+		psql $CONN_STRING -c "truncate $TABLENAME;"
+	fi
+done
+
+echo "$(date): $0 COMPLETED"
+exit 0
\ No newline at end of file
diff --git a/tools/hedgehog/database_update_geoip.in b/tools/hedgehog/database_update_geoip.in
new file mode 100755
index 0000000..acc0dd8
--- /dev/null
+++ b/tools/hedgehog/database_update_geoip.in
@@ -0,0 +1,86 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_update_geoip
+#
+
+# We need a write user
+a=$(@LIBEXEC@/check_user_perms -wc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+TMP_DIR=/tmp
+
+usage () {
+	echo
+	echo "Update the GeopIP list in the database. This reads input from ${TMP_DIR}"
+	echo 
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+}
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+cd $TMP_DIR
+rm -f GeoLite2-City-CSV.zip.md5
+curl -O http://geolite.maxmind.com/download/geoip/database/GeoLite2-City-CSV.zip.md5
+if [ -f GeoLite2-City-CSV.zip ] ; then
+  md5sum GeoLite2-City-CSV.zip | awk ' { print $1 } ' | diff -w GeoLite2-City-CSV.zip.md5 -
+  if [ $? -ne 0 ] ; then
+    rm GeoLite2-City-CSV.zip
+  fi
+fi
+if [ ! -f GeoLite2-City-CSV.zip ] ; then
+  curl -O http://geolite.maxmind.com/download/geoip/database/GeoLite2-City-CSV.zip
+  md5sum GeoLite2-City-CSV.zip | awk ' { print $1 } ' | diff -w GeoLite2-City-CSV.zip.md5 -
+  if [ $? -ne 0 ] ; then
+    echo "Error downloading and md5-ing GeoLite2-City-CSV.zip"
+    exit 1
+  fi
+fi
+  
+unzip -oj GeoLite2-City-CSV.zip
+
+echo "Preparing City Blocks data..."
+@LIBEXEC@/utility_prepare_GeoLite2-City-Blocks-IPvX
+
+echo "Preparing Location data..."
+@LIBEXEC@/utility_prepare_GeoLite2-City-Locations-en
+			
+psql $CONN_STRING << EOF                                                                                                                                          
+BEGIN;
+delete from geoip;
+\copy geoip (ip_range, geoname, latitude, longitude) from '/tmp/blocks_ready_to_import.csv' (HEADER false, DELIMITER ',', FORMAT 'csv')
+delete from locations;
+\copy locations (geoname_id, name, city) from '/tmp/locations_ready_to_import.csv' (HEADER false, DELIMITER ',', FORMAT 'csv')
+COMMIT;
+EOF
+
+rm -f /tmp/blocks_ready_to_import.csv /tmp/locations_ready_to_import.csv
+
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/database_update_nodes.in b/tools/hedgehog/database_update_nodes.in
new file mode 100755
index 0000000..86d6ba8
--- /dev/null
+++ b/tools/hedgehog/database_update_nodes.in
@@ -0,0 +1,299 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_update_nodes
+#
+
+eval $(@LIBEXEC@/conf_read)
+NODES_FILE="${hhconfig_directories_conf}/nodes.csv"
+DATA_DIR=$hhconfig_directories_data
+if [ ! -e $NODES_FILE ] ; then
+	echo "Error attempting to find nodes.csv. Tried: $NODES_FILE"
+	exit 1
+fi
+
+# We need a write user with access to the data dir. 
+a=$(@LIBEXEC@/check_user_perms -wfc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+usage () {
+    echo
+    echo "Add/update the servers and nodes in the database. This reads input from file"
+    echo "called nodes.csv in the $hhconfig_directories_conf directory."
+    echo " - An example nodes.csv file is installed if one does not exist."
+    echo " - See the comments in that file for details of the format."
+    echo " - No action is taken on servers/nodes that are in the database but are not in the input files."
+    echo 
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -r rename/remove flag     This enables processing of the rename_or_remove field in nodes.csv"
+    echo "                            If this flag is not given any lines in the input file that have"
+    echo "                            text in the rename_or_remove field are ignored."
+    echo "                            Renaming a node will update the database and move the data "
+    echo "                            directory to the new name."
+    echo "                            Removing a node will delete all associated data in the database"
+    echo "                            but leave any data on disk untouched (for manual removal)."
+    echo "                            Removing nodes could take a long time if you have a lot of data."
+    echo "  -d debug mode (set -x)."
+    echo "  -h Show this help."
+}
+RENAME=0
+while getopts ":rhd" opt; do
+    case $opt in
+        r  ) RENAME=1 ;;
+        h  ) usage
+             exit 1 ;;
+        d  ) set -x ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+# Read CSV file with the following fields in order:
+fields=(server_name node_name node_region node_country node_city node_subgroup node_ip old_name)
+
+NAME_REGEX="^[a-zA-Z0-9\\.-]+$"
+NAME_SPACE_REGEX="^[[:space:]a-zA-Z0-9\\-]+$"
+NAME_ERROR="It may contain alphanumeric characters, hyphens (-) and full stops (.) only."
+NAME_SPACE_ERROR="It may contain alphanumeric characters, spaces and hyphens (-) only."
+IP_REGEXP="^[a-fA-F0-9:\\.]*$"
+IP_ERROR="It must be a valid IPv4 or IPv6 address."
+# This restriction comes from the maximum table name length.
+MAX_SERVERNAME_LEN=11
+
+
+declare -A aNODES
+declare -A aFIELD_OPTIONAL
+declare -A aFIELD_REGEX
+declare -A aFIELD_ERROR
+
+aFIELD_OPTIONAL["server_name"]="no"
+aFIELD_OPTIONAL["node_name"]="no"
+aFIELD_OPTIONAL["node_region"]="yes"
+aFIELD_OPTIONAL["node_country"]="yes"
+aFIELD_OPTIONAL["node_city"]="yes"
+aFIELD_OPTIONAL["node_subgroup"]="yes"
+aFIELD_OPTIONAL["node_ip"]="yes"
+aFIELD_OPTIONAL["old_name"]="yes"
+
+aFIELD_REGEX["server_name"]=$NAME_REGEX
+aFIELD_REGEX["node_name"]=$NAME_REGEX
+aFIELD_REGEX["node_region"]=$NAME_REGEX
+aFIELD_REGEX["node_country"]=$NAME_SPACE_REGEX
+aFIELD_REGEX["node_city"]=$NAME_SPACE_REGEX
+aFIELD_REGEX["node_subgroup"]=$NAME_REGEX
+aFIELD_REGEX["node_ip"]=$IP_REGEXP
+aFIELD_REGEX["old_name"]=$NAME_REGEX
+
+aFIELD_ERROR["server_name"]=$NAME_ERROR
+aFIELD_ERROR["node_name"]=$NAME_ERROR
+aFIELD_ERROR["node_region"]=$NAME_ERROR
+aFIELD_ERROR["node_country"]=$NAME_SPACE_ERROR
+aFIELD_ERROR["node_city"]=$NAME_SPACE_ERROR
+aFIELD_ERROR["node_subgroup"]=$NAME_ERROR
+aFIELD_ERROR["node_ip"]=$IP_ERROR
+aFIELD_ERROR["old_name"]=$NAME_ERROR
+
+i=0
+NEW_SERVER=0
+NODES_UPDATED=0
+NODES_ADDED=0
+NODES_ERRORS=0
+NODES_RENAMED=0
+NODES_REMOVED=0
+echo
+echo "PROCESSING:   SERVER, NODE, GROUP, COUNTRY, CITY, INSTANCE, IP ADDRESS, OLD NAME"
+# Read nodes.csv
+while read line || [[ -n "$line" ]]; do
+	i=$(($i+1))
+	trimmed_line=$(echo $line | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
+
+	if [[ -z ${trimmed_line} ]] || [[ "${trimmed_line}" =~ ^# ]] ; then
+		continue
+	fi	 
+
+	IFS=, read -r aNODES["server_name"] aNODES["node_name"] \
+	                aNODES["node_region"] aNODES["node_country"] \
+	                aNODES["node_city"] aNODES["node_subgroup"] \
+	                aNODES["node_ip"] aNODES["old_name"] \
+	                junk <<< "$trimmed_line"
+	
+	echo
+	if [[ -n "$junk" ]] ; then
+		echo "***ERROR***:  Line $i, Too many fields. This line will be ignored."
+		((NODES_ERRORS++))
+		continue
+	fi
+
+	error="no"
+	# Turn off filename expansion in case someone has put something stupid 
+	# in the nodes file
+	set -f
+	for key in ${fields[*]} ; do
+		aNODES["$key"]=$(echo ${aNODES["$key"]} | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
+		if [[ ${aNODES["$key"]} =~ ^[[:space:]]*$ ]] ; then
+			# Empty IP address MUST be null not an empty string
+			if [ "$key" == "node_ip" ] ; then
+				aNODES["$key"]="null"
+			fi
+			if [ "${aFIELD_OPTIONAL["$key"]}" == "no" ] ; then
+				echo "***ERROR***:  Line $i, Field $key is not optional."
+				((NODES_ERRORS++))
+				error="yes"
+				break
+			else
+				continue
+			fi
+		fi
+		if [ $RENAME -eq 0 ] && [ -n "${aNODES["old_name"]}" ] ; then
+			echo "***ERROR***:  Line $i, old-name field is not empty: contains \"${aNODES["old_name"]}\" but -r (rename) flag is not set."
+			((NODES_ERRORS++))
+			error="yes"
+			break
+		fi
+		if [[ ! ${aNODES["$key"]} =~ ${aFIELD_REGEX["$key"]} ]] ; then
+			echo "***ERROR***:  Line $i, Field $key value: \"${aNODES["$key"]}\" is not valid. ${aFIELD_ERROR["$key"]} This line will be ignored."
+			((NODES_ERRORS++))
+			error="yes"
+			break
+		fi
+	done
+	if [ "$error" == "yes" ] ; then
+		continue
+	fi
+	set +f
+	
+	echo -n "PROCESSING:   ${aNODES["server_name"]}, ${aNODES["node_name"]}, ${aNODES["node_region"]}, ${aNODES["node_country"]}, ${aNODES["node_city"]}, ${aNODES["node_subgroup"]}"
+	if [ ${aNODES["node_ip"]} != "null" ] ; then
+		echo -n ", ${aNODES["node_ip"]}"
+	else
+		echo -n ","
+	fi
+	if [ -n "${aNODES["old_name"]}" ] ; then
+		echo ", ${aNODES["old_name"]}"
+	else
+		echo ","
+	fi
+	
+	SERVER_ID=$(psql $CONN_STRING -tc "select id from server where display_name='${aNODES["server_name"]}'")
+	if [ -z "$SERVER_ID" ]; then
+		# lower case
+		SERVER_DB_NAME=${aNODES["server_name"]}
+		SERVER_DB_NAME=${SERVER_DB_NAME,,}
+		# subsitute _ for -
+		SERVER_DB_NAME=${SERVER_DB_NAME//-/_}
+		# subsitute __ for .
+		SERVER_DB_NAME=${SERVER_DB_NAME//./__}
+		if [ ${#SERVER_DB_NAME} -gt $MAX_SERVERNAME_LEN ] ; then
+			echo "***ERROR***:  Line $i, The Server name ${aNODES["server_name"]} is too long to be stored internally in Hedgehog."
+			echo "              Names containing only alphanumeric characters and hyphens can be a maximum of $MAX_SERVERNAME_LEN characters long."
+			echo "              Names also containing full stops must be less than this. This line will be ignored."
+			((NODES_ERRORS++))
+			continue
+		fi
+		SERVER_ID=$(psql $CONN_STRING -tqc "INSERT INTO dsc.server (name, display_name) VALUES ('$SERVER_DB_NAME', '${aNODES["server_name"]}') RETURNING id;")
+		echo "ADDED SERVER: ${aNODES["server_name"]}"
+		((NEW_SERVER++))
+	fi
+	
+	if [ $RENAME -eq 1 ] && [ -n "${aNODES["old_name"]}" ] ; then
+		NODE_ID=$(psql $CONN_STRING -tc "select id from dsc.node where server_id=$SERVER_ID and name='${aNODES["old_name"]}'")
+		if [ "${aNODES["old_name"]}" == "remove" ] ; then
+			NODE_ID=$(psql $CONN_STRING -tc "select id from dsc.node where server_id=$SERVER_ID and name='${aNODES["node_name"]}'")
+			if [ -n "$NODE_ID" ] ; then
+				psql $CONN_STRING -c "BEGIN; DELETE FROM dsc.data where node_id = ${NODE_ID}; DELETE FROM node where id = ${NODE_ID}; COMMIT;"
+				((NODES_REMOVED++))
+				echo "The data directory for Server:${aNODES["server_name"]} Node: ${aNODES["node_name"]} can now be manually removed."
+				continue
+			else
+				echo "***ERROR***:  Line $i, The node called ${aNODES["node_name"]} could not be found"
+				((NODES_ERRORS++))
+				continue
+			fi
+		elif [ -z "$NODE_ID" ] ; then
+			echo "***ERROR***:  Line $i, The node called ${aNODES["old_name"]} could not be found"
+			((NODES_ERRORS++))
+			continue
+		else
+			psql $CONN_STRING -c "UPDATE dsc.node SET name='${aNODES["node_name"]}' WHERE id=${NODE_ID};"
+			((NODES_RENAMED++))
+		fi
+	else
+		NODE_ID=$(psql $CONN_STRING -tc "select id from dsc.node where server_id=$SERVER_ID and name='${aNODES["node_name"]}'")
+	fi
+	if [ ${aNODES["node_ip"]} != "null" ] ; then
+		aNODES["node_ip"]="'${aNODES["node_ip"]}'"
+	fi
+	if [ -z "$NODE_ID" ] ; then
+		NODE_ID=$(psql $CONN_STRING -tqc "INSERT INTO dsc.node (server_id, name, region, country, city, subgroup, ip) VALUES ($SERVER_ID, '${aNODES["node_name"]}', '${aNODES["node_region"]}', '${aNODES["node_country"]}', '${aNODES["node_city"]}', '${aNODES["node_subgroup"]}', ${aNODES["node_ip"]});")
+		echo "ADDED NODE:   ${aNODES["server_name"]}, ${aNODES["node_name"]}"
+		((NODES_ADDED++))
+	else
+		# It would be nice here to only update when something has changed so the user can easily see what changed
+		psql $CONN_STRING -c "UPDATE dsc.node SET server_id=$SERVER_ID, name='${aNODES["node_name"]}', region='${aNODES["node_region"]}', country='${aNODES["node_country"]}', city='${aNODES["node_city"]}', subgroup='${aNODES["node_subgroup"]}', ip=${aNODES["node_ip"]} where id=$NODE_ID;"
+		echo "UPDATED NODE: ${aNODES["server_name"]}, ${aNODES["node_name"]}"
+		((NODES_UPDATED++))
+	fi
+	if [ $RENAME -eq 1 ] && [ -n "${aNODES["old_name"]}" ] && [ "${aNODES["old_name"]}" != "remove" ] ; then
+		if [ ! -d $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]} ] ; then
+			echo "Error: Unable to find $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]}."
+			((NODES_ERRORS++))
+			error="yes"
+			break
+		elif [ -d $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]} ] ; then
+			echo "Error: New directory $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]} already exists."
+			((NODES_ERRORS++))
+			error="yes"
+			break
+		else
+			mv -vT $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]} $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]}
+			if [ $? -ne 0 ] ; then
+				echo "Error: Failed to move $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]}."
+				((NODES_ERRORS++))
+				error="yes"
+				break
+			fi
+		fi
+	else
+		mkdir -p $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]}/incoming
+		chmod 775 $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]}/incoming
+	fi
+	junk=""
+done < ${NODES_FILE}
+
+	echo 
+	echo "SUMMARY:"
+	echo " - Servers added    : $NEW_SERVER"
+	echo " - Nodes added      : $NODES_ADDED"
+	echo " - Nodes updated    : $NODES_UPDATED"
+	echo " - Nodes renamed    : $NODES_RENAMED"
+	echo " - Nodes removed    : $NODES_REMOVED"
+	echo " - Nodes with errors: $NODES_ERRORS"
+	echo
+
+if [ ${NEW_SERVER} -ge 1 ] ; then
+	echo "********************************************************************************"
+	echo "New servers have been added to Hedgehog! To enable the import of data for this"
+	echo "server the 'manage_partitions' script MUST be run manually now."
+	echo "********************************************************************************"
+	echo
+fi
+
+echo "$(date): $0 COMPLETED"
\ No newline at end of file
diff --git a/tools/hedgehog/database_update_service_addr.in b/tools/hedgehog/database_update_service_addr.in
new file mode 100755
index 0000000..3673706
--- /dev/null
+++ b/tools/hedgehog/database_update_service_addr.in
@@ -0,0 +1,96 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_update_service_addr
+#
+
+eval $(@LIBEXEC@/conf_read)
+SERVICE_ADDR_FILE="${hhconfig_directories_conf}/service_addr.csv"
+if [ ! -e $SERVICE_ADDR_FILE ] ; then
+	echo "Error attempting to find service_addr.csv. Tried: $SERVICE_ADDR_FILE"
+	exit 1
+fi
+
+# We need a write user
+a=$(@LIBEXEC@/check_user_perms -wc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+DELETE=0
+
+usage () {
+    echo
+    echo "Update the Server service addreses list in the database."
+    echo "This list is only used to filter which addresses are displayed in the"
+    echo "'By server IP address' plot. If there are any addresses in the list"
+    echo "then only these addresses are displayed. If there are no entries "
+    echo "in the list then all the addresses for which there is data are displayed."
+    echo "By default this script reads the $SERVICE_ADDR_FILE file and adds"
+    echo "the addresses found there."
+    echo "After running this script it is advisable to remove any cached plots"
+    echo "for the 'By server IP address' plot by using the plotcache_rm_cached_plots"
+    echo "script."
+    echo 
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -d Delete all existing entries in the server service address list"
+    echo "  -h Show this help."
+}
+
+while getopts ":dh" opt; do
+    case $opt in
+        d  ) DELETE=1 ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+if [ ${DELETE} -eq 1 ] ; then
+    psql $CONN_STRING << EOF
+DELETE FROM service_addr;
+EOF
+echo "$(date): $0 COMPLETED"
+exit 0
+fi
+
+i=0
+while read line || [[ -n "$line" ]]; do
+    i=$(($i+1))
+    trimmed_line=$(echo $line | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
+
+    if [[ -z ${trimmed_line} ]] || [[ "${trimmed_line}" =~ ^# ]] ; then
+            continue
+    fi
+    IFS=, read -r SERVER_NAME ADDRESS junk <<< "$trimmed_line"
+    if [[ -n $junk ]] ; then
+            echo "***ERROR***:  Line $i, Too many fields. This line will be ignored."
+            continue
+    fi
+    echo $SERVER_NAME -- $ADDRESS
+psql $CONN_STRING << EOF
+BEGIN;
+insert into service_addr (server_id, address) select id, '$ADDRESS' from server where display_name = '$SERVER_NAME';
+COMMIT;
+EOF
+    junk=""
+done < ${SERVICE_ADDR_FILE}
+
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/database_update_tlds.in b/tools/hedgehog/database_update_tlds.in
new file mode 100755
index 0000000..e2944b0
--- /dev/null
+++ b/tools/hedgehog/database_update_tlds.in
@@ -0,0 +1,103 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_update_tlds
+#
+
+# We need a write user
+a=$(@LIBEXEC@/check_user_perms -wc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+TLDS_DIR=$CONF_DIR/tlds.d
+TLD_TMP="/tmp/hedgehog_tld_inserts.sql"
+COUNT=0
+
+usage () {
+	echo
+	echo "Update the TLD list in the database. This reads input from files"
+	echo "from the $TLDS_DIR directory."
+	echo "No action is taken on TLDs that are in the database but are not in the input files."
+	echo 
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+}
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+if [ ! -d "$TLDS_DIR" ] ; then
+	echo "No tlds.d dir found. Expecting a dir $TLDS_DIR... done..."
+	exit 1
+fi
+
+echo "insert into tlds (alabel, type)" > $TLD_TMP
+
+for TLD_FILE in ${TLDS_DIR}/* ; do
+  # Process the tld file
+  echo "Processing file: " $TLD_FILE
+  while read line || [[ -n "$line" ]]; do
+  	trimmed_line=$(echo $line | sed -e 's/^ *//' -e 's/ *$//')
+
+  	if [[ -z ${trimmed_line} ]] || [[ "${trimmed_line}" =~ ^# ]] ; then
+  		continue
+  	fi	
+	
+  	OIFS=$IFS
+  	IFS=','
+  	read -ra arrline <<< "$trimmed_line"
+  	IFS=$OIFS
+  	ALABEL=$(echo ${arrline[0]} | sed -e 's/^ *//' -e 's/ *$//')
+    TLDTYPE=$(echo ${arrline[1]} | sed -e 's/^ *//' -e 's/ *$//')
+    
+  	# First, validate the input	
+  	if [ -z "$ALABEL" ] ; then
+  		echo "Missing alabel. This entry will be ignored."
+  		continue
+  	fi	
+	
+  	if ! [[ "${ALABEL}" =~ ^[a-zA-Z0-9\-]+$ ]] ; then
+  		echo "alabel name '${ALABEL}' is invalid. It may contain alphanumeric characters and hyphens (-) only. This entry will be ignored."
+  		continue
+  	fi
+    
+  	if [ -z "$TLDTYPE" ] ; then
+    	echo "Missing TLD type field for alabel $ALABEL - alabel ignored"
+    	continue
+    fi
+    
+    if [ $COUNT -gt 0 ]; then
+      echo "UNION" >> $TLD_TMP
+    fi
+    echo "select lower('${ALABEL}'), id from tld_types where type_name='${TLDTYPE}'" >> $TLD_TMP
+    (( COUNT+=1 ))
+  done < ${TLD_FILE}
+done
+echo ";" >> $TLD_TMP
+psql $CONN_STRING -tf $TLD_TMP
+
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/database_update_tlds_from_zone.in b/tools/hedgehog/database_update_tlds_from_zone.in
new file mode 100755
index 0000000..9171771
--- /dev/null
+++ b/tools/hedgehog/database_update_tlds_from_zone.in
@@ -0,0 +1,64 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_update_tlds_from_zone
+#
+
+# We need a write user
+a=$(@LIBEXEC@/check_user_perms -wc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+TLD_TMP="/tmp/hedgehog_tld_inserts_from_zone.sql"
+
+usage () {
+	echo
+	echo "Update the TLD list in the database. This reads input from IANA ftp site"
+	echo "from the $TLDS_DIR directory."
+	echo "No action is taken on TLDs that are in the database but are not in the input data."
+	echo 
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -h Show this help."
+}
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+rm $TLD_TMP
+
+TTYPE=$(psql $CONN_STRING -tc "select id from tld_types where type_name='New-gTLD'")
+TCOUNT1=$(psql $CONN_STRING -tc "select count(*) from tlds where type = $TTYPE")
+
+for tld in $(curl ftp://rs.internic.net/domain/root.zone | awk -F\. ' { if (NF == 2) print tolower($1) } ' | awk '/./' | sort | uniq ) ; do
+
+    echo "insert into tlds (alabel, type) values ('$tld', $TTYPE);" >> $TLD_TMP
+
+done
+psql $CONN_STRING -tf $TLD_TMP
+TCOUNT2=$(psql $CONN_STRING -tc "select count(*) from tlds where type = $TTYPE")
+TNEW=$((TCOUNT2 - TCOUNT1))
+echo "Added $TNEW New gTLDs to database."
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/datafiles_create_summary.in b/tools/hedgehog/datafiles_create_summary.in
new file mode 100755
index 0000000..313e856
--- /dev/null
+++ b/tools/hedgehog/datafiles_create_summary.in
@@ -0,0 +1,170 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   datafiles_create_summary
+#
+
+# We need user with access to the database
+a=$(@LIBEXEC@/check_user_perms -c)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+RED='\033[0;31m'
+LOG_FILE="dsc-extractor-xml-db.out"
+NODE_NAME=""
+SERVER_NAME=""
+FORMAT_STRING="----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------"
+FORMAT_SEP="|"
+
+print_header () {
+	echo $FORMAT_STRING
+	echo " Server    $FORMAT_SEP Node      $FORMAT_SEP Last import started $FORMAT_SEP Last file processed $FORMAT_SEP Files found $FORMAT_SEP Files remaining $FORMAT_SEP Failures $FORMAT_SEP Last file processed                          $FORMAT_SEP Time      $FORMAT_SEP Error found in last"
+	echo "           $FORMAT_SEP           $FORMAT_SEP                     $FORMAT_SEP at                  $FORMAT_SEP in incoming $FORMAT_SEP in import job   $FORMAT_SEP          $FORMAT_SEP                                              $FORMAT_SEP taken (s) $FORMAT_SEP 20 lines of file"
+	echo $FORMAT_STRING
+}
+
+usage () {
+	echo
+	echo "Generate a processing report based on the information in the log files"
+	echo 
+	echo "Usage: $(basename $0) options"
+	echo
+	echo "Supported options:"
+	echo "	-s Only report on the server with this name (default is all servers)"
+	echo "	-n Only report on the node with this name (default is all nodes)"
+	echo "	-c Output in csv format for import into spreadsheet application"
+	echo "	-d Run in debug mode (set -x)"
+	echo "	-h Show this help."
+}
+
+while getopts ":s:n:cdh" opt; do
+    case $opt in
+		n  ) NODE_NAME=$OPTARG ;;
+		s  ) SERVER_NAME=$OPTARG ;;
+		d  ) set -x ;;
+		c  ) FORMAT_STRING=""
+		     FORMAT_SEP=',';;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+TOTAL_IN_PROGRESS=0
+
+if [ -z "${SERVER_NAME}" ] ; then
+	SERVER_NAMES=$(psql $CONN_STRING -tc "select display_name from dsc.server")
+else 
+	SERVER_NAMES=$SERVER_NAME
+fi
+
+print_header
+
+MIDNIGHT=$(date --date="00:00 today" +%s)
+
+declare -a PROCESSING_NODES
+
+for SERVER in $SERVER_NAMES ; do
+
+	test -L $DATADIR/$SERVER && continue;
+	test -d $DATADIR/$SERVER || continue;
+
+	SERVER_ID=$(psql $CONN_STRING -tc "select id from dsc.server where display_name='${SERVER}'")
+
+	if [ -z "${NODE_NAME}" ] ; then
+		NODE_NAMES=$(psql $CONN_STRING -tc "select name from dsc.node where server_id=$SERVER_ID")
+	else
+		NODE_NAMES=$NODE_NAME
+	fi
+	COUNTER=0
+	for NAME in $NODE_NAMES  ; do
+
+		FILE_NAME=$DATADIR/$SERVER/$NAME/$LOG_FILE
+		if [ ! -f "${FILE_NAME}" ] ; then
+			continue
+		fi
+
+		if [ ! -d "$DATADIR/$SERVER/$NAME/incoming/" ] ; then
+			continue
+		fi
+
+		if [ ! "$(ls -A $DATADIR/$SERVER/$NAME/incoming/)" ] ; then
+			continue
+		fi
+
+		COUNTER=$(( COUNTER + 1 ))
+		# Get the last date that import started:
+		IMPORT_START_TIME=$(tac ${FILE_NAME} | grep -e 'UTC ----------------------$' -m1 | awk '{print $2, $3}')
+		COMPLETED_LINE=$(tac ${FILE_NAME} | grep -e 'Completed' -m1 | awk '{print $1, $2, $5, $6, $8, $9}')
+		IMPORT_LATEST_TIME=$(echo $COMPLETED_LINE | awk '{print $1, $2}')
+
+		FOUND_FILES=$(find ${DATADIR}/${SERVER}/${NAME}/incoming/*/ -name \*.xml | wc -l)
+		FILES_REMAINING_NUMBER=$(echo $COMPLETED_LINE | awk '{print $4}')
+		FILES_FOUND_NUMBER=$(echo $FOUND_FILES | awk '{print $1}' )
+		FILES_FAILED=$(echo $COMPLETED_LINE | sed 's/(//'  | awk '{print $5}')
+
+		LAST_FILE_PATH=$(echo $COMPLETED_LINE | awk '{print $3}')
+		LAST_FILE_NAME=$(basename "$LAST_FILE_PATH")
+		LAST_FILE_UNIX_TIME="${LAST_FILE_NAME%.*.*}"
+		if [ -z $LAST_FILE_UNIX_TIME ] ; then
+			LAST_FILE=""
+		else
+			LAST_FILE=$(date -d @${LAST_FILE_UNIX_TIME} "+%Y-%m-%d %H:%M:%S")
+		        LAST_FILE_START_TIME=$(tac ${FILE_NAME} |  grep -e "Starting:  ${LAST_FILE_PATH}"  -m 1 | awk '{print $1 " " $2}')
+			LAST_FILE_FINISH_TIME=$(echo $COMPLETED_LINE | awk '{print $1 " " $2}')
+			LAST_FILE_START_UNIX_TIME=$(date --date="${LAST_FILE_START_TIME}" +"%s")
+			LAST_FILE_FINISH_UNIX_TIME=$(date --date="${LAST_FILE_FINISH_TIME}" +"%s")
+			DURATION=$(( LAST_FILE_FINISH_UNIX_TIME - LAST_FILE_START_UNIX_TIME))
+		fi
+		(( TOTAL_FILES_FOUND = TOTAL_FILES_FOUND + FILES_FOUND_NUMBER ))
+
+		ERR_COUNT=$(tail ${FILE_NAME} -n 20 | grep  -i -e 'Error' | grep  -v 'lock pid file' | wc -l)
+		if [ $ERR_COUNT -ne 0 ] ; then
+			ERROR_STR="YES"
+		else
+			ERROR_STR="NO"
+		fi
+
+		if [ $FILES_REMAINING_NUMBER -gt 0 ] && [ $ERROR_STR == "NO" ] ; then
+			(( TOTAL_IN_PROGRESS = TOTAL_IN_PROGRESS + 1 ))
+			(( TOTAL_PROCESSING_TIME = TOTAL_PROCESSING_TIME + DURATION ))
+			PROCESSING_NODES+=" "${SERVER}/${NAME}
+		fi
+		
+		if [ $LAST_FILE_UNIX_TIME -lt $MIDNIGHT ] ; then
+			# color red
+			printf '\033[0;31m%-10s %-1s %-9s %-1s %-16s %-1s %-19s %-1s %-11s %-1s %-15s %-1s %-8s %-1s %-19s %-22s %-1s %-9s %-1s %-15s\033[0m\n'  "$SERVER" $FORMAT_SEP  "$NAME" $FORMAT_SEP "${IMPORT_START_TIME}" $FORMAT_SEP "$IMPORT_LATEST_TIME" $FORMAT_SEP "$FOUND_FILES" $FORMAT_SEP "$FILES_REMAINING_NUMBER" $FORMAT_SEP  "$FILES_FAILED" $FORMAT_SEP "$LAST_FILE" "($LAST_FILE_NAME)" $FORMAT_SEP "$DURATION" $FORMAT_SEP "${ERROR_STR}"
+		elif [ $FOUND_FILES -gt 720 ] ; then
+			# color yellow
+			printf '\033[0;33m%-10s %-1s %-9s %-1s %-16s %-1s %-19s %-1s %-11s %-1s %-15s %-1s %-8s %-1s %-19s %-22s %-1s %-9s %-1s %-15s\033[0m\n'  "$SERVER" $FORMAT_SEP  "$NAME" $FORMAT_SEP "${IMPORT_START_TIME}" $FORMAT_SEP "$IMPORT_LATEST_TIME" $FORMAT_SEP "$FOUND_FILES" $FORMAT_SEP "$FILES_REMAINING_NUMBER" $FORMAT_SEP  "$FILES_FAILED" $FORMAT_SEP "$LAST_FILE" "($LAST_FILE_NAME)" $FORMAT_SEP "$DURATION" $FORMAT_SEP "${ERROR_STR}"
+		else
+			# no color
+			printf '%-10s %-1s %-9s %-1s %-16s %-1s %-19s %-1s %-11s %-1s %-15s %-1s %-8s %-1s %-19s %-22s %-1s %-9s %-1s %-15s\n'  "$SERVER" $FORMAT_SEP  "$NAME" $FORMAT_SEP "${IMPORT_START_TIME}" $FORMAT_SEP "$IMPORT_LATEST_TIME" $FORMAT_SEP "$FOUND_FILES" $FORMAT_SEP "$FILES_REMAINING_NUMBER" $FORMAT_SEP  "$FILES_FAILED" $FORMAT_SEP "$LAST_FILE" "($LAST_FILE_NAME)" $FORMAT_SEP "$DURATION" $FORMAT_SEP "${ERROR_STR}"
+		fi
+
+		if [ $COUNTER -eq 21 ] ; then
+			print_header
+			COUNTER=0
+		fi
+	done
+done
+print_header
+
+echo "Time: " $(date "+%Y-%m-%d %H:%M:%S") " Total files remaining to import: " $TOTAL_FILES_FOUND
+PERF=$(( TOTAL_PROCESSING_TIME / TOTAL_IN_PROGRESS ))
+echo "Time: " $(date "+%Y-%m-%d %H:%M:%S") " Processors in progress: " ${TOTAL_IN_PROGRESS} " at approx " ${PERF} " seconds per datafile"
+echo "List of nodes currently processing: $PROCESSING_NODES"
diff --git a/tools/hedgehog/datafiles_rm_empty_xml_dirs.in b/tools/hedgehog/datafiles_rm_empty_xml_dirs.in
new file mode 100755
index 0000000..a47420c
--- /dev/null
+++ b/tools/hedgehog/datafiles_rm_empty_xml_dirs.in
@@ -0,0 +1,101 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   datafiles_rm_empty_xml_dirs.bash
+#
+
+# We need user with access to the data directory
+a=$(@LIBEXEC@/check_user_perms -fnc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+
+
+DATE=$(date --date="- 7 day" +%Y-%m-%d)
+INCOMING="incoming"
+
+usage () {
+	echo
+	echo "Remove empty incoming xml directories older than a certain date (default is all the processed directories older than 7 days ago)"
+	echo 
+	echo "Usage: $(basename $0) options"
+	echo
+	echo "Supported options:"
+	echo "	-d Date before which to remove empty xml directories (default: 7 days ago, format YYYY-MM-DD)"
+	echo "	-h Show this help."
+}
+
+while getopts ":d:h" opt; do
+    case $opt in
+        d  ) DATE=$OPTARG ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+[ -z "$DATE" ] && usage && exit 1
+
+# Validate the start date
+if [[ ! $DATE =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]$ ]] ; then
+		echo "Error: The start date is not in the required format of YYYY-MM-DD" 
+		exit 1
+fi
+date "+%Y-%m-%d" --date="$DATE" >/dev/null 2>&1
+is_valid=$?
+if [ ${is_valid} -ne 0 ] ; then
+	echo "Error: The start date is invalid" 
+	exit 1
+fi
+
+# Date converted in seconds to be compared with processed directories date
+DATE_SECONDS=$(date --utc --date "$DATE" +%s)
+
+cd $DATADIR
+
+for SERVER in * ; do
+	test -L $SERVER && continue;
+	test -d $SERVER || continue;
+	cd $DATADIR/$SERVER
+	for NODE in * ; do
+		test -L $NODE && continue;
+		test -d $NODE || continue;
+		test -L $NODE/$DONE && continue;
+        test -d $NODE/$DONE || continue;
+        cd $DATADIR/$SERVER/$NODE/$INCOMING
+		for MYDATE in * ; do
+			test -L $MYDATE && continue;
+			test -d $MYDATE || continue;
+			date "+%Y-%m-%d" -d $MYDATE >/dev/null 2>&1
+			is_valid=$?
+			if [ ${is_valid} -ne 0 ] ; then
+				continue;
+			fi
+			# Check if directory is older than DATE
+			MYDATE_SECONDS=$(date --utc --date "$MYDATE" +%s)
+			[ $MYDATE_SECONDS -ge $DATE_SECONDS ] && continue;
+			# Remove empty directories
+			my_count=$(ls -1 $MYDATE | wc -l)
+			[ $my_count -eq 0 ] && rm -rf $MYDATE
+		done
+		cd $DATADIR/$SERVER
+	done
+	cd $DATADIR
+done
+
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/datafiles_tar_old_xml_files.in b/tools/hedgehog/datafiles_tar_old_xml_files.in
new file mode 100755
index 0000000..5a4ba54
--- /dev/null
+++ b/tools/hedgehog/datafiles_tar_old_xml_files.in
@@ -0,0 +1,53 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# We need user with access to the data directory
+a=$(@LIBEXEC@/check_user_perms -fnc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+
+usage () {
+  echo
+  echo "Pack old xml files already processed into a .tar package to store and archive."
+  echo "The xml files are packed according to their date and node and server."
+  echo "The filename has the following format done-PID-date.tar.bz2 (e.g. done-14143-2015-05-01.tar.bz2)."
+  echo 
+  echo "Usage: $(basename $0) options"
+  echo
+  echo "Supported options:"
+  echo "  -h Show this help."
+  exit 1
+}
+
+while getopts ":h" opt; do
+    case $opt in
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+for i in $(find $DATADIR -maxdepth 3 -name "done" -type d) ; do
+    for j in $(find $i -mindepth 1 -maxdepth 1 -type d) ; do
+    short=$(echo $j | sed "s|$DATADIR||")
+    DIR_FOUND=$(echo $short | awk -F\/ ' { print $5 } ')
+    TODAY=$(date +%Y-%m-%d)
+    if [ $DIR_FOUND == $TODAY ] ; then
+      continue
+    else
+      echo "tar -cj --remove-files -f $i/done-$$-$DIR_FOUND.tar.bz2 $j"
+      nice -n 19 tar -cj --remove-files -f $i/done-$$-$DIR_FOUND.tar.bz2 $j
+    fi
+  done
+done
+
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/hedgehog_database_create b/tools/hedgehog/hedgehog_database_create
new file mode 100755
index 0000000..bd95802
--- /dev/null
+++ b/tools/hedgehog/hedgehog_database_create
@@ -0,0 +1,184 @@
+#!/bin/bash
+# 
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   database_create
+#
+
+DB_NAME=hedgehog
+DB_OWNER=hedgehog
+DB_READER=www-data
+
+usage () {
+	echo
+	echo "Create database and roles/users."
+	echo
+	echo "Usage: $(basename $0) options"
+	echo
+	echo "Supported options:"
+	echo "  -o Owner of the database  (Default: $DB_OWNER)"
+	echo "  -r Reader of the database (Default: $DB_READER) Typically the same user that the web server runs as."
+	echo "  -n Name of the database   (Default: $DB_NAME)"
+	echo "  -h Show this help"
+	echo
+	echo "In the 'database' section of hedgehog.yaml you should set the same values"
+  
+  exit 1
+}
+
+while getopts ":o:r:n:h" opt; do
+    case $opt in
+        o  ) DB_OWNER=$OPTARG ;;
+        r  ) DB_READER=$OPTARG ;;
+        n  ) DB_NAME=$OPTARG ;;
+        h  ) usage ;;
+        \? ) usage ;;
+    esac
+done
+
+DB_EXISTS=$(psql -tlq | cut -d \| -f 1 | grep -w $DB_NAME)
+[ "x$DB_EXISTS" != "x" ] && echo "The $DB_NAME Database exists. If you want to overwrite it then drop it manually and re-run this script" && exit 1
+
+psql <<EOF
+CREATE USER "$DB_OWNER";
+CREATE ROLE hh_read;
+CREATE ROLE hh_write;
+CREATE DATABASE "$DB_NAME"
+  WITH OWNER = "$DB_OWNER"
+  ENCODING = 'UTF8'
+  TABLESPACE = pg_default
+  CONNECTION LIMIT = -1;
+CREATE USER "$DB_READER";
+GRANT hh_read TO "$DB_READER";
+ALTER USER "$DB_READER" SET search_path TO dsc;
+ALTER USER "$DB_OWNER" SET search_path TO dsc;
+\connect "$DB_NAME"
+CREATE LANGUAGE plpythonu;
+CREATE SCHEMA dsc AUTHORIZATION "$DB_OWNER";
+CREATE EXTENSION ip4r SCHEMA dsc;
+
+
+CREATE or REPLACE FUNCTION dsc.ip2asn (addr dsc.ipaddress)
+  RETURNS text
+as \$\$
+
+  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
+  if rdata != None:
+    result = rdata[0]['teamcymrulookup']
+    if result != None:
+      return result.split('|')[0].strip()
+          
+  plpy.info('Function ip2asn: No ASN found')
+  return None
+\$\$ LANGUAGE plpythonu;
+
+CREATE or REPLACE FUNCTION dsc.ip2bgpprefix (addr dsc.ipaddress)
+  RETURNS int
+as \$\$
+
+  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
+  if rdata != None:
+    result = rdata[0]['teamcymrulookup']
+    if result != None:
+      bgpprefix = result.split('|')[1].strip()
+      return int(bgpprefix.split('/')[1])
+          
+  plpy.info('Function ip2bgpprefix: No BGP Prefix found')
+  return None
+\$\$ LANGUAGE plpythonu;
+
+CREATE or REPLACE FUNCTION dsc.teamcymrulookup (addr dsc.ipaddress)
+  RETURNS text
+as \$\$
+  from IPy import IP
+  import getdns
+  
+  ''' Sanity Check '''
+  if addr is None:
+    plpy.error('Function teamcymrulookup: Address passed in was null')
+    return None
+    
+  ''' Create a IPy IP object '''
+  ip = IP(addr)
+
+  ''' reverse the address '''
+  if ip.version() == 4:
+    s = ip.strFullsize()
+    s = s.split('.')
+    s.reverse()
+    s = '.'.join(s)
+    qname = "%s.origin.asn.cymru.com." % s
+  elif ip.version() == 6:
+    s = '%032x' % ip.int()
+    s = list(s)
+    s.reverse()
+    s = '.'.join(s)
+    qname = "%s.origin6.asn.cymru.com." % s 
+  else:
+    plpy.error('Function teamcymrulookup: Unknown IP address type.')
+    return None
+    
+  ''' Store/retrieve the context in/from the global dict. '''
+  if 'teamcymrulookupctx' in GD.keys():
+      ctx = GD['teamcymrulookupctx']
+  else:
+      ctx = getdns.Context()
+      ctx.resolution_type = getdns.RESOLUTION_STUB
+      GD['teamcymrulookupctx'] = ctx
+  
+  try:
+    results = ctx.general(name=qname, request_type=getdns.RRTYPE_TXT, extensions={})
+  except getdns.error as e:
+    plpy.error('Function teamcymrulookup: GetDNS Error: %s' % str(e))
+    return None
+  status = results.status
+  if status == getdns.RESPSTATUS_GOOD:
+    for reply in results.replies_tree:
+      answers = reply['answer']
+      for answer in answers:
+        if answer['type'] == getdns.RRTYPE_TXT:
+          rdata = answer['rdata']
+          return rdata['txt_strings'][0]
+          
+  plpy.info('Function teamcymrulookup: No data returned from DNS query')
+  return None
+\$\$ LANGUAGE plpythonu;
+
+CREATE or REPLACE FUNCTION dsc.iptruncate (addr dsc.ipaddress)
+  RETURNS dsc.iprange
+as \$\$
+BEGIN
+  CASE family(addr)
+    WHEN 4 then RETURN addr / 8;
+    WHEN 6 then RETURN addr / 32;
+  END CASE;
+END
+\$\$ LANGUAGE plpgsql;
+
+CREATE or REPLACE FUNCTION dsc.alabel2ulabel (alabel text)
+  RETURNS text
+as \$\$
+  
+  import getdns
+  
+  ''' Sanity Check '''
+  if alabel is None:
+    plpy.error('Function alabel2ulabel: a-label passed in was null')
+    return None
+  
+  ulabel = getdns.alabel_to_ulabel(alabel)
+  return ulabel
+\$\$ LANGUAGE plpythonu;
+EOF
+
+echo
+echo "Please add $DB_OWNER and $DB_READER to pg_hba.conf, set passwords etc."
diff --git a/tools/hedgehog/plotcache_generate_cached_plots.in b/tools/hedgehog/plotcache_generate_cached_plots.in
new file mode 100755
index 0000000..e3163d4
--- /dev/null
+++ b/tools/hedgehog/plotcache_generate_cached_plots.in
@@ -0,0 +1,133 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   plotcache_generate_cached_plots
+#
+
+# We need user with access to the plotcache directory
+a=$(@LIBEXEC@/check_user_perms -pc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+FROM=$(date --date="- 1 day" +%Y-%m-%dT00:00)
+TO=$(date --date="- 1 day" +%Y-%m-%dT23:59)
+SQ="'"
+
+usage () {
+    echo
+    echo "Create cached plots for each chosen period (day/week/month) between Start and End range of time"
+    echo "You must add at least one option for which plot window to generate [-D|-W|-M]"
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -s Start date and time for which to create cached plots (default: \"${FROM}\")"
+    echo "  -e End date and time for which to create cached plots (default: \"${TO}\")"
+    echo "  -D Generate daily cached plots"
+    echo "  -W Generate weekly cached plots"
+    echo "  -M Generate monthly cached plots"
+    echo "  -h Show this help."
+}
+
+while getopts ":s:e:DWMh" opt; do
+    case $opt in
+        s  ) FROM=$OPTARG ;;
+        e  ) TO=$OPTARG ;;
+        D  ) DAY=1 ;;
+        W  ) WEEK=1 ;;
+        M  ) MONTH=1 ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+# Validate the start date
+if [[ ! $FROM =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]$ ]] ; then
+		echo "Error: The start date is not in the required format of YYYY-MM-DDThh:mm" 
+		exit 1
+fi
+date "+%Y-%m-%dT%H:%M" --date="${FROM}" >/dev/null 2>&1
+is_valid=$?
+if [ ${is_valid} -ne 0 ] ; then
+	echo "Error: The start date is invalid" 
+	exit 1
+fi
+
+# Validate the end date
+if [[ ! $TO =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]$ ]] ; then
+		echo "Error: The end date is not in the required format of YYYY-MM-DDThh:mm" 
+		exit 1
+fi
+date "+%Y-%m-%dT%H:%M" --date="${TO}" >/dev/null 2>&1
+is_valid=$?
+if [ ${is_valid} -ne 0 ] ; then
+	echo "Error: The end date is invalid" 
+	exit 1
+fi
+
+# Check that Start date is earlier than End date
+FROM_seconds=$(date --utc --date "${FROM}" +%s)
+TO_seconds=$(date --utc --date "${TO}" +%s)
+if [ ${FROM_seconds} -ge ${TO_seconds} ] ; then
+  echo "Error: Start date must be earlier than End date to generate plots"
+  exit 1
+fi
+
+if [[ -z "$DAY" && -z "$WEEK" && -z "$MONTH" ]] ; then
+  usage
+  exit 1
+fi
+
+echo "INFO: $(find $PLOT_CACHE_DIR -type f | wc -l) cached plots"
+echo "SYSMON INFO: $(du -c $PLOT_CACHE_DIR | grep total) disk usage for $PLOT_CACHE_DIR"
+
+# TODO: do not hard code this path
+URL="localhost/hedgehog/brew/hedgehog.brew"
+TYPE="generatePlot"
+declare -a STOPS=($(seq $(date +%s -d ${TO}) -86400 $(date +%s -d ${FROM})))
+INTERACTIVE="0 1"
+PLOTIDS=$(psql $CONN_STRING -tc "select id from dsc.plot where name in (select name from dsc.visible_plot) ORDER BY id DESC;")
+SERVERNMS=$(psql $CONN_STRING -tc "SELECT display_name FROM dsc.server;")
+NDARR=-1 # all nodes
+
+# Call brew script in order to cache the default plots
+for SVRNM in $SERVERNMS ; do
+    SVRID=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server WHERE display_name=${SQ}${SVRNM}${SQ};" | tr -d [[:space:]])
+    for STOP in ${STOPS[@]} ; do
+        STOP=$(date --date @${STOP} +%Y-%m-%dT%R)
+        STARTS=()
+        [ -n "$DAY" ] && STARTS+=($(date --date="${STOP}" +%Y-%m-%dT00:00))
+        [ -n "$WEEK" ] && STARTS+=($(date --date="${STOP} 1 week ago + 1 day" +%Y-%m-%dT00:00))
+        [ -n "$MONTH" ] && STARTS+=($(date --date="${STOP} 1 month ago + 1 day" +%Y-%m-%dT00:00))
+        for START in ${STARTS[@]} ; do
+            for GVIS in $INTERACTIVE ; do
+                for PLTID in $PLOTIDS ; do
+                    echo $(date) ": curl --max-time 300 --retry 1 --get \"${URL}?type=${TYPE}&start=${START}&stop=${STOP}&gvis=${GVIS}&pltid=${PLTID}&svrnm=${SVRNM}&svrid=${SVRID}&ndarr=${NDARR}\""
+                    curl -s -o /dev/null --max-time 300 --retry 1 --get "${URL}?type=${TYPE}&start=${START}&stop=${STOP}&gvis=${GVIS}&pltid=${PLTID}&svrnm=${SVRNM}&svrid=${SVRID}&ndarr=${NDARR}" 2>&1
+                done
+            done
+        done
+    done
+done
+
+echo "INFO: $(find $PLOT_CACHE_DIR -type f | wc -l) cached plots"
+echo "SYSMON INFO: $(du -c $PLOT_CACHE_DIR | grep total) disk usage for $PLOT_CACHE_DIR"
+echo "$(date): $0 COMPLETED"
\ No newline at end of file
diff --git a/tools/hedgehog/plotcache_rm_cached_plots.in b/tools/hedgehog/plotcache_rm_cached_plots.in
new file mode 100755
index 0000000..ef7b357
--- /dev/null
+++ b/tools/hedgehog/plotcache_rm_cached_plots.in
@@ -0,0 +1,98 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   plotcache_rm_cached_plots
+#
+
+# We need user with access to the plotcache directory
+a=$(@LIBEXEC@/check_user_perms -pc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+REMOVE_ALL=0
+
+usage () {
+    echo
+    echo "This script is interactive if the -a option is not selected."
+    echo "You will be asked to select which cached plot types you'd like to remove"
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -a Remove all cached plots"
+    echo "  -h Show this help."
+}
+
+while getopts ":ah" opt; do
+    case $opt in
+        a  ) REMOVE_ALL=1 ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+if [ $REMOVE_ALL -eq 1 ] ; then
+	echo "Removing all cached plots from $PLOT_CACHE_DIR"
+	rm -rf $PLOT_CACHE_DIR/*/*/*.png
+	rm -rf $PLOT_CACHE_DIR/*/*/*.html
+	echo "Done."
+	exit
+fi
+
+# Get list of servers and plots from database
+SERVERS=$(psql $CONN_STRING -tc "SELECT display_name FROM dsc.server;")
+PLOTS=$(psql $CONN_STRING -tc "SELECT name FROM dsc.visible_plot;")
+PLOTS=(${PLOTS// / }) # convert string to an array
+
+while true ; do
+    if [ ${#PLOTS[@]} -eq 0 ] ; then
+        exit
+    fi
+    echo "Please select which cached plot type you'd like to clear or q to exit:"
+    echo
+    n=0
+    for plot in ${PLOTS[@]} ; do
+        let n=n+1
+        echo "    ${n}. ${plot}"
+    done
+    echo
+    echo -n "Enter selection or q to exit: "
+    read c
+    if [ "$c" == "Q" ] || [ "$c" == "q" ] ; then
+        exit 1
+    elif [[ $c -ge 1 ]] && [[ $c -le $n ]] ; then
+        let c=c-1
+        selection=${PLOTS[${c}]}
+        PLOTS=(${PLOTS[@]:0:$c} ${PLOTS[@]:$(($c + 1))}) # remove selected plot from PLOTS
+        for server in $SERVERS ; do
+            echo "rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.png"
+            rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.png
+            echo "rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.html"
+            rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.html
+            echo "Removed plots for ${selection}."
+        done
+    else
+        echo "invalid selection - please try again or exit by typing 'q'"
+    fi
+	echo "Do you want to remove another plot?"
+done
+
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/rssac_generate_reports.in b/tools/hedgehog/rssac_generate_reports.in
new file mode 100755
index 0000000..cbe4873
--- /dev/null
+++ b/tools/hedgehog/rssac_generate_reports.in
@@ -0,0 +1,139 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# 
+# File:   rssac_generate_reports
+#
+
+#TODO(asap): Add a flag to allow the user to specify a single server to run the report for
+
+# eval $(@LIBEXEC@/conf_read)
+
+a=$(@LIBEXEC@/check_user_perms -pgc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+DEFAULT_DAY=$(date --date="- 7 day" +%Y-%m-%d)
+START=$(date --date="${DEFAULT_DAY}" +%Y-%m-%dT00:00)
+INPUT_START=""
+
+usage () {
+    echo
+    echo "Create RSSAC reports for a specified 24 hour period"
+    echo "(Requires data to have been processed using the database_process_rssac_data script)"
+    echo
+    echo "Usage: $(basename $0) options"
+    echo
+    echo "Supported options:"
+    echo "  -d Date on which to create RSSAC reports (default is 7 days ago: \"${DEFAULT_DAY}\", format: YYYY-MM-DD)"
+    echo "  -h Show this help"
+}
+
+while getopts ":d:h" opt; do
+    case $opt in
+        d  ) INPUT_START=$OPTARG ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+echo "$(date): $0 COMMENCED"
+
+# Validate the input date
+if [ ! -z "${INPUT_START}" ] ; then
+	if [[ ! $INPUT_START =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]$ ]] ; then
+			echo "Error: The start date is not in the required format of YYYY-MM-DD" 
+			exit 1
+	fi
+	date "+%Y-%m-%d" --date="${INPUT_START}" >/dev/null 2>&1
+	is_valid=$?
+	if [ ${is_valid} -ne 0 ] ; then
+		echo "Error: The start date is invalid" 
+		exit 1
+	fi
+	START=$(date --date="${INPUT_START}" +%Y-%m-%dT00:00)
+        DEFAULT_DAY=$(date --date="${INPUT_START}" +%Y-%m-%d)
+fi
+
+STOP=$(date --date="${START}" +%Y-%m-%dT23:59)
+
+SQ="'"
+CURL_ARGS="--max-time 300 --retry 1 --get"
+# TODO: do not hardcode this path
+URL="localhost/hedgehog/brew/hedgehog.brew"
+TOP_OUTPUTDIR="$RSSAC_DIR/$(date --date=${START} +%Y)/$(date --date=${START} +%m)"
+TYPES="generateYaml generatePlot"
+GVIS=0
+# Specify some strings for special handling
+TRAFFIC_SIZE_SMALL="traffic_sizes_small"
+TRAFFIC_SIZE_BIG="traffic_sizes_big"
+TRAFFIC_SIZE_REP_NAME="traffic-sizes"
+TRAFFIC_VOLUME_DIFF="traffic_volume_difference"
+
+echo
+echo "Creating RSSAC reports from ${START} to ${STOP} in ${TOP_OUTPUTDIR}" 
+echo
+
+PLOTIDS=$(psql $CONN_STRING -tc "SELECT id FROM dsc.plot where ddcategory='9.RSSAC' and name != '$TRAFFIC_VOLUME_DIFF' ORDER BY id DESC;")
+SERVERNMS=$(psql $CONN_STRING -tc "SELECT display_name FROM dsc.server;")
+NDARR=-1 # all nodes
+
+for SVRNM in $SERVERNMS ; do
+    SVRID=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server WHERE display_name=${SQ}${SVRNM}${SQ};" | tr -d [[:space:]])
+    SVRNM_LC="$(echo ${SVRNM} | tr '[:upper:]' '[:lower:]')"
+
+    for TYPE in $TYPES ; do
+        EXT="yaml"
+        if [ "${TYPE}" == "generatePlot" ] ; then
+			EXT="png"
+		fi
+		
+        for PLTID in $PLOTIDS ; do
+			PLTNM=$(psql $CONN_STRING -tc "SELECT name FROM dsc.plot where id=${PLTID};" | sed 's/[[:space:]]//g')
+			# Clear out the file in the plot cache to make sure we get the most up to date plots in the reports
+			rm -rf $RSSAC_DIR/${SVRNM}/${PLTNM}/${START}-${STOP}-all*.png >/dev/null 2>&1
+			# Generate the file. Yaml gets written to the rssac directory, plots go into the plotcache
+			echo "*** Generating '${EXT}' for: " ${PLTNM}
+			CURL_URL="${URL}?type=${TYPE}&start=${START}&stop=${STOP}&gvis=${GVIS}&pltid=${PLTID}&svrnm=${SVRNM}&svrid=${SVRID}&ndarr=${NDARR}"
+            echo $(date) ": curl $CURL_ARGS \"$CURL_URL\""
+            RESULT=$(curl $CURL_ARGS "$CURL_URL" 2>&1)
+            echo ${RESULT}
+			echo
+			
+            if [ "${TYPE}" == "generatePlot" ] ; then
+				# Retrieve the plot from the plot cache
+                PLTFILE=$(echo ${RESULT} | sed 's/^.* src="\([^"?]*\).*/\1/' | sed "s:../plotcache:$PLOT_CACHE_DIR:" | sed 's/-log//' | sed 's/-stack//')
+
+                if [ "${PLTFILE}" != "plots/no_graph.png" ] && [ "${PLTFILE}" != "plots/no_results.png" ] && [ "${PLTFILE}" != "plots/no_connection.png" ] ; then                   
+					# Report names are the same as plot names but use dash instead of underscore
+					# TODO: use $PLTNM instead of redoing the query then substitute _ with -
+					REP_NAME=$(psql $CONN_STRING -tc "SELECT name FROM dsc.plot where id=${PLTID};" | sed 's/_/-/g' | sed 's/[[:space:]]//g')
+					# Special handling. The traffic size data has 2 plots but both need to go into the same directory
+					PLT_DIR_NM=${REP_NAME}
+					if [ "${PLTNM}" == "${TRAFFIC_SIZE_SMALL}" ] || [ "${PLTNM}" == "${TRAFFIC_SIZE_BIG}" ] ; then
+						PLT_DIR_NM="${TRAFFIC_SIZE_REP_NAME}"
+					fi
+                    OUTPUTDIR="${TOP_OUTPUTDIR}/${PLT_DIR_NM}/"
+                    OUTPUTFILE="${SVRNM_LC}-`date --date=${START} +%Y%m%d`-${REP_NAME}.png"
+                    cp ${PLTFILE} ${OUTPUTDIR}${OUTPUTFILE}
+                fi
+            fi
+        done
+    done
+done
+echo "$(date): $0 COMPLETED"
diff --git a/tools/hedgehog/sql/dns.sql b/tools/hedgehog/sql/dns.sql
new file mode 100644
index 0000000..f1c4c76
--- /dev/null
+++ b/tools/hedgehog/sql/dns.sql
@@ -0,0 +1,127 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* Take an IPv[46] address and look it up in the Team Cymru IP to ASN DNS service
+ * See http://www.team-cymru.org/IP-ASN-mapping.html
+ */
+
+CREATE or REPLACE FUNCTION dsc.ip2asn (addr dsc.ipaddress)
+  RETURNS text
+as $$
+
+  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
+  if rdata != None:
+    result = rdata[0]['teamcymrulookup']
+    if result != None:
+      return result.split('|')[0].strip()
+          
+  plpy.info('Function ip2asn: No ASN found')
+  return None
+$$ LANGUAGE plpythonu;
+
+CREATE or REPLACE FUNCTION dsc.ip2bgpprefix (addr dsc.ipaddress)
+  RETURNS int
+as $$
+
+  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
+  if rdata != None:
+    result = rdata[0]['teamcymrulookup']
+    if result != None:
+      bgpprefix = result.split('|')[1].strip()
+      return int(bgpprefix.split('/')[1])
+          
+  plpy.info('Function ip2bgpprefix: No BGP Prefix found')
+  return None
+$$ LANGUAGE plpythonu;
+
+CREATE or REPLACE FUNCTION dsc.teamcymrulookup (addr dsc.ipaddress)
+  RETURNS text
+as $$
+  from IPy import IP
+  import getdns
+  
+  ''' Sanity Check '''
+  if addr is None:
+    plpy.error('Function teamcymrulookup: Address passed in was null')
+    return None
+    
+  ''' Create a IPy IP object '''
+  ip = IP(addr)
+
+  ''' reverse the address '''
+  if ip.version() == 4:
+    s = ip.strFullsize()
+    s = s.split('.')
+    s.reverse()
+    s = '.'.join(s)
+    qname = "%s.origin.asn.cymru.com." % s
+  elif ip.version() == 6:
+    s = '%032x' % ip.int()
+    s = list(s)
+    s.reverse()
+    s = '.'.join(s)
+    qname = "%s.origin6.asn.cymru.com." % s 
+  else:
+    plpy.error('Function teamcymrulookup: Unknown IP address type.')
+    return None
+    
+  ''' Store/retrieve the context in/from the global dict. '''
+  if 'getdnsctx' in GD.keys():
+      ctx = GD['getdnsctx']
+  else:
+      ctx = getdns.Context()
+      ctx.resolution_type = getdns.RESOLUTION_STUB
+      GD['getdnsctx'] = ctx
+  
+  try:
+    results = ctx.general(name=qname, request_type=getdns.RRTYPE_TXT, extensions={})
+  except getdns.error as e:
+    plpy.error('Function teamcymrulookup: GetDNS Error: %s' % str(e))
+    return None
+  status = results.status
+  if status == getdns.RESPSTATUS_GOOD:
+    for reply in results.replies_tree:
+      answers = reply['answer']
+      for answer in answers:
+        if answer['type'] == getdns.RRTYPE_TXT:
+          rdata = answer['rdata']
+          return rdata['txt_strings'][0]
+          
+  plpy.info('Function teamcymrulookup: No data returned from DNS query')
+  return None
+$$ LANGUAGE plpythonu;
+
+CREATE or REPLACE FUNCTION dsc.iptruncate (addr dsc.ipaddress)
+  RETURNS dsc.iprange
+as $$
+BEGIN
+  CASE family(addr)
+    WHEN 4 then RETURN addr / 8;
+    WHEN 6 then RETURN addr / 32;
+  END CASE;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE or REPLACE FUNCTION dsc.alabel2ulabel (alabel text)
+  RETURNS text
+as $$
+  
+  import getdns
+  
+  ''' Sanity Check '''
+  if alabel is None:
+    plpy.error('Function alabel2ulabel: a-label passed in was null')
+    return None
+  
+  ulabel = getdns.alabel_to_ulabel(alabel)
+  return ulabel
+$$ LANGUAGE plpythonu;
diff --git a/tools/hedgehog/sql/initialise_database.sql b/tools/hedgehog/sql/initialise_database.sql
new file mode 100755
index 0000000..b933d85
--- /dev/null
+++ b/tools/hedgehog/sql/initialise_database.sql
@@ -0,0 +1,305 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+-----------------
+--  DSC SCHEMA --
+-----------------
+
+--
+-- Name: version; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.version (
+    version integer
+);
+
+--
+-- Name: internal_version; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.internal_version (
+    serial integer,
+    script character varying(255),
+    description character varying(255),
+    applied timestamp with time zone
+);
+
+--
+-- Name: server; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.server
+(
+  id serial NOT NULL,
+  name character varying(255) NOT NULL,
+  display_name character varying(255)  NOT NULL,
+  description character varying(255)
+)
+WITH (
+  OIDS=FALSE
+);
+
+--
+-- Name: node; Type: TABLE; Schema: dsc; 
+--
+--TODO(asap): Just have server_id, name and group
+CREATE TABLE dsc.node
+(
+  id serial NOT NULL,
+  server_id integer NOT NULL,
+  name character varying(255) NOT NULL,
+  city character varying(255),
+  state character varying(255),
+  country character varying(255),
+  region character varying(255) NOT NULL
+)
+WITH (
+  OIDS=FALSE
+);
+
+--
+-- Name: dataset; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.dataset
+(
+  id serial NOT NULL,
+  name character varying(255) NOT NULL
+)
+WITH (
+  OIDS=FALSE
+);
+
+--
+-- Name: visible_plot; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.visible_plot
+(
+  id serial NOT NULL, 
+  name character varying(255) NOT NULL	,
+  display_category character varying(255),
+  display_name character varying(255),
+  title character varying(255),
+  description character varying(255)
+)
+WITH (
+  OIDS=FALSE
+);
+
+--
+-- Name: dataset_visible_plot; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.dataset_visible_plot
+(
+  visible_plot_id integer NOT NULL,
+  dataset_id integer NOT NULL
+)
+WITH (
+  OIDS=FALSE
+);
+
+--
+-- Name: plot; Type: VIEW; Schema: dsc; 
+-- This view is used for backwards compatibility and to provide an interface
+-- into the new dataset and visible plots tables since there is now a 
+-- many to many relationship between them. When the front end code is 
+-- refactored it can be replaced by a cleaner view!
+--
+CREATE VIEW dsc.plot AS
+(
+	SELECT ROW_NUMBER() OVER(ORDER BY name) AS id, * FROM
+	(
+		SELECT
+			vp.name,
+			vp.display_category AS ddcategory,
+			vp.display_name AS ddname,
+			vp.title,
+			vp.description,
+			(SELECT dataset_id FROM dataset_visible_plot WHERE dsc.dataset_visible_plot.visible_plot_id=vp.id LIMIT 1) AS plot_id,
+			(SELECT ARRAY(SELECT dataset_id FROM dataset_visible_plot WHERE dsc.dataset_visible_plot.visible_plot_id=vp.id)) AS dataset_id
+		FROM
+			dsc.visible_plot AS vp
+		UNION
+		SELECT
+			name,
+			'' AS ddcategory,
+			'' AS ddname,
+			'' AS title,
+			'' AS description,
+			id AS plot_id,
+			ARRAY[id] AS dataset_id
+		FROM
+			dataset
+		WHERE
+			dsc.dataset.name NOT IN (SELECT name FROM dsc.visible_plot)
+	)
+	AS x
+);
+
+--
+-- Name: query_classification; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.query_classification (
+    id serial NOT NULL,
+    name character varying(255),
+    title character varying(255)
+);
+
+--
+-- Name: iana_lookup; Type: TABLE; Schema: dsc; 
+--
+CREATE TABLE dsc.iana_lookup
+(
+  registry character varying(255)  NOT NULL,
+  value integer NOT NULL,
+  name character varying(255) NOT NULL,
+  description character varying(255)
+)
+WITH (
+  OIDS=FALSE
+);
+
+--
+-- Name: data; Type: TABLE; Schema: dsc; 
+--
+--TODO(refactor): Strictly don't need the server_id here. 
+CREATE TABLE dsc.data
+(
+  starttime timestamp with time zone NOT NULL,
+  server_id integer NOT NULL,
+  node_id integer NOT NULL,
+  plot_id integer NOT NULL,
+  key1 character varying(255) NOT NULL,
+  key2 character varying(255) NOT NULL,
+  value integer NOT NULL
+)
+WITH (
+  OIDS=FALSE
+);
+
+--
+-- Name: pk_server, uniq_server; Type: CONSTRAINT; Schema: dsc; 
+--
+ALTER TABLE ONLY dsc.server
+  ADD CONSTRAINT pk_server PRIMARY KEY (id),
+  ADD CONSTRAINT uniq_server UNIQUE (name);
+
+--
+-- Name: pk_nodes, fk_nodes_servers, uniq_nodes; Type: CONSTRAINT; Schema: dsc; 
+--
+ALTER TABLE ONLY dsc.node
+  ADD CONSTRAINT pk_node PRIMARY KEY (id),
+  ADD CONSTRAINT fk_node_server FOREIGN KEY (server_id) REFERENCES dsc.server (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION,
+  ADD CONSTRAINT uniq_node UNIQUE(server_id, name);
+
+--
+-- Name: pk_dataset, uniq_dataset_name; Type: CONSTRAINT; Schema: dsc; 
+--
+ALTER TABLE ONLY dsc.dataset
+  ADD CONSTRAINT pk_dataset PRIMARY KEY (id),
+  ADD CONSTRAINT uniq_dataset_name UNIQUE (name);
+
+--
+-- Name: pk_visible_plot, uniq_visible_plot_name; Type: CONSTRAINT; Schema: dsc; 
+--
+ALTER TABLE ONLY dsc.visible_plot
+  ADD CONSTRAINT pk_visible_plot PRIMARY KEY (id),
+  ADD CONSTRAINT uniq_visible_plot_name UNIQUE (name);
+
+
+--
+-- Name: pk_il; Type: CONSTRAINT; Schema: dsc; 
+--
+ALTER TABLE ONLY dsc.iana_lookup
+  ADD CONSTRAINT pk_il PRIMARY KEY (registry , value , name );
+
+--
+-- Name: pk_data, fk_data_node, fk_data_plot; Type: CONSTRAINT; Schema: dsc; 
+--
+ALTER TABLE dsc.data
+  ADD CONSTRAINT pk_data PRIMARY KEY (server_id, node_id, plot_id, starttime, key1, key2),
+  ADD CONSTRAINT fk_data_node FOREIGN KEY (node_id) REFERENCES dsc.node (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION,
+  ADD CONSTRAINT fk_data_plot FOREIGN KEY (plot_id) REFERENCES dsc.dataset(id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION;
+
+/*
+--
+-- Name: idx_data; Type: INDEX; Schema: dsc; 
+--
+CREATE INDEX idx_data
+  ON dsc.data
+  USING btree
+  (plot_id COLLATE pg_catalog."default" );
+*/
+
+--
+-- Name: fn_dsc_data_insert_trigger; Type: FUNCTION; Schema: dsc; 
+--
+CREATE OR REPLACE FUNCTION dsc.fn_dsc_data_insert_trigger()
+RETURNS TRIGGER AS $$
+DECLARE
+    server TEXT;
+    plot TEXT;
+    tbl_nm TEXT;
+    ins_sql TEXT;
+BEGIN
+    SELECT name INTO STRICT server FROM dsc.server WHERE id = new.server_id;
+    SELECT name INTO STRICT plot FROM dsc.dataset WHERE id = new.plot_id;
+    tbl_nm := 'data_' || server || '_' || plot || '_' || to_char(NEW.starttime, 'YYYY_MM');
+    ins_sql := 
+        'INSERT INTO dsc.'|| tbl_nm
+        || ' (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES'
+        || ' ('
+        || quote_literal(NEW.starttime) || ','
+        || NEW.server_id || ','
+        || NEW.node_id || ','
+        || NEW.plot_id || ','
+        || quote_literal(NEW.key1) || ','
+        || quote_literal(NEW.key2) || ','
+        || NEW.value
+        || ')'
+        ;
+    EXECUTE ins_sql; 
+    RETURN NULL;
+EXCEPTION
+    WHEN undefined_table THEN
+        RAISE WARNING 'dsc data has not been inserted due to undefined partition --> %', tbl_nm;
+        RETURN NULL;
+END
+$$
+LANGUAGE 'plpgsql' ;
+
+--
+-- Name: dsc_data_insert_trigger; Type: TRIGGER; Schema: dsc; 
+--
+CREATE TRIGGER dsc_data_insert_trigger
+    BEFORE INSERT ON dsc.data
+    FOR EACH ROW EXECUTE PROCEDURE dsc.fn_dsc_data_insert_trigger();
+
+--
+-- Name: unique_source_summary_function(integer, integer, timestamp with time zone, timestamp with time zone, text); Type: FUNCTION; Schema: dsc; 
+--
+CREATE FUNCTION unique_source_summary_function(integer, integer, timestamp with time zone, timestamp with time zone, text) RETURNS TABLE(x character varying, y bigint)
+    LANGUAGE plpgsql
+    AS $_$
+BEGIN
+return query SELECT 
+				key1 as x, 
+				count(key2) AS y 
+				FROM dsc.data as d 
+				WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) 
+				GROUP BY x 
+			UNION 
+			SELECT 
+				'IPv6/64' as x, 
+				count(*) as y from (SELECT substring(key2 FROM '(^([0-9a-f]{1,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}))') as subnet 
+				FROM dsc.data as d 
+				WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1='IPv6' AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY subnet) AS sq 
+				ORDER BY y DESC;
+END
+$_$;
+
diff --git a/tools/hedgehog/sql/insert_lookup_data.sql b/tools/hedgehog/sql/insert_lookup_data.sql
new file mode 100755
index 0000000..4cce861
--- /dev/null
+++ b/tools/hedgehog/sql/insert_lookup_data.sql
@@ -0,0 +1,396 @@
+/* 
+ * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+ * 
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, you can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+/*
+ * Developed by Sinodun IT (www.sinodun.com)
+ */
+
+/* Both are bumped from 2.0 version by 1 since DDL 10 is not run on fresh install.
+   This is because it requires super user priviledges so is done
+   as part of database_create. */
+INSERT INTO dsc.version (version) VALUES 
+    (8);
+
+INSERT INTO dsc.internal_version ( serial, script, description, applied ) VALUES
+    (10, '-', 'DB creation', now() );
+
+INSERT INTO dataset (name) VALUES ('certain_qnames_vs_qtype');
+INSERT INTO dataset (name) VALUES ('chaos_types_and_names');
+INSERT INTO dataset (name) VALUES ('client_addr_vs_rcode_accum');
+INSERT INTO dataset (name) VALUES ('client_subnet2_accum');
+INSERT INTO dataset (name) VALUES ('client_subnet2_count');
+INSERT INTO dataset (name) VALUES ('client_subnet2_trace');
+INSERT INTO dataset (name) VALUES ('client_subnet_accum');
+INSERT INTO dataset (name) VALUES ('client_subnet_count');
+INSERT INTO dataset (name) VALUES ('direction_vs_ipproto');
+INSERT INTO dataset (name) VALUES ('dns_ip_version');
+INSERT INTO dataset (name) VALUES ('dns_ip_version_vs_qtype');
+INSERT INTO dataset (name) VALUES ('dnssec_qtype');
+INSERT INTO dataset (name) VALUES ('do_bit');
+INSERT INTO dataset (name) VALUES ('edns_version');
+INSERT INTO dataset (name) VALUES ('idn_qname');
+INSERT INTO dataset (name) VALUES ('idn_vs_tld');
+INSERT INTO dataset (name) VALUES ('ipv6_rsn_abusers_accum');
+INSERT INTO dataset (name) VALUES ('ipv6_rsn_abusers_count');
+INSERT INTO dataset (name) VALUES ('opcode');
+INSERT INTO dataset (name) VALUES ('pcap_stats');
+INSERT INTO dataset (name) VALUES ('qtype');
+INSERT INTO dataset (name) VALUES ('qtype_vs_qnamelen');
+INSERT INTO dataset (name) VALUES ('qtype_vs_tld');
+INSERT INTO dataset (name) VALUES ('rcode');
+INSERT INTO dataset (name) VALUES ('rcode_vs_replylen');
+INSERT INTO dataset (name) VALUES ('rd_bit');
+INSERT INTO dataset (name) VALUES ('traffic_sizes_queries');
+INSERT INTO dataset (name) VALUES ('traffic_sizes_responses');
+INSERT INTO dataset (name) VALUES ('traffic_volume_queries');
+INSERT INTO dataset (name) VALUES ('traffic_volume_responses');
+INSERT INTO dataset (name) VALUES ('transport_vs_qtype');
+INSERT INTO dataset (name) VALUES ('unique_sources');
+INSERT INTO dataset (name) VALUES ('unique_sources_raw');
+
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('by_node', '1.Node Statistics', 'By node', 'Queries by node');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('chaos_types_and_names', '2.Query Attributes', 'CHAOS queries', 'CHAOS queries');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('do_bit', '2.Query Attributes', 'DO bit', 'Queries with the DNSSEC OK bit set');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('edns_version', '2.Query Attributes', 'EDNS version', 'EDNS version');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('idn_qname', '2.Query Attributes', 'IDN qnames', 'Queries containing internationalized qnames');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('opcode', '2.Query Attributes', 'OPCODE', 'Breakdown of OPCODE values, other than QUERY');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rd_bit', '2.Query Attributes', 'RD bit', 'Queries with Recursion Desired bit set');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('qtype', '3.QTYPE', 'DNS queries by QTYPE', 'QTYPE values in DNS queries');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('dnssec_qtype', '3.QTYPE', 'DNSSEC queries by QTYPE', 'QTYPE values in DNSSEC queries');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('certain_qnames_vs_qtype', '3.QTYPE', 'Popular query names by QTYPE', 'Queries for the name localhost and anything under root-servers.net.');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('qtype_vs_tld', '3.QTYPE', 'QTYPE for most popular TLDs', 'QTYPE values for most popular TLDs queried');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('qtype_vs_qnamelen', '3.QTYPE', 'Query name lengths by QTYPE', 'Query name lengths showing QTYPE');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode', '4.RCODE', 'Replies by RCODE', 'RCODE values in DNS replies');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode_vs_replylen', '4.RCODE', 'Reply lengths by RCODE (<1000 bytes)', 'Size of DNS replies showing RCODE for message size below 1000 bytes');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode_vs_replylen_big', '4.RCODE', 'Reply lengths by RCODE (>=1000 bytes)', 'Size of DNS replies showing RCODE for message size above 1000 bytes');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('dns_ip_version', '5.IP Protocol', 'IP version', 'IP version carrying DNS queries');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('dns_ip_version_vs_qtype', '5.IP Protocol', 'Queries by IP version and QTYPE', 'QTYPE by IP version');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('direction_vs_ipproto', '5.IP Protocol', 'Received packets by IP protocol', 'Recieved Packets by IP protocol');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('transport_vs_qtype', '5.IP Protocol', 'Transports carrying DNS queries', 'Transports carrying DNS queries');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('pcap_stats', '6.PCAP Statistics', 'PCAP Statistics', 'Packet capture statistics');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet_accum', '7.Client Subnet Statistics', 'Busiest client subnets', 'Busiest client subnets (IPv4/8 or IPv6/32)');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_addr_vs_rcode_accum', '7.Client Subnet Statistics', 'RCODE by client subnet', 'Busiest client subnets (IPv4/8 or IPv6/32) showing RCODE');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('ipv6_rsn_abusers_accum', '7.Client Subnet Statistics', 'Root abusers', 'Clients sending excessive root-servers.net queries');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet2_accum', '8.Classification', 'Query classification by client subnet (accum)', 'Query classification by client subnet (IPv4/8 or IPv6/32)');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet2_count', '8.Classification', 'Query classification by client subnet (count)', 'Count of client subnet sending each query classification');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet2_trace', '8.Classification', 'Query classifications', 'Queries by classification');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode_volume', '9.RSSAC', 'RCODE volume', 'Count of rcodes in responses');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_sizes_small', '9.RSSAC', 'Traffic sizes (<1000 bytes)', 'Histogram of DNS query and response sizes by transport for message size below 1000 bytes');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_sizes_big', '9.RSSAC', 'Traffic sizes (>=1000 bytes)', 'Histogram of DNS query and response sizes by transport for message size above 1000 bytes');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_volume', '9.RSSAC', 'Traffic volume', 'The number of queries and responses by transport and IP version');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('unique_sources', '9.RSSAC', 'Unique sources', 'The number of unique source addresses');
+INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_volume_difference', '9.RSSAC', 'Traffic volume difference', 'The differences between queries and responses by transport and IP version');
+
+
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='by_node'), (SELECT id FROM dataset WHERE name='qtype'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='chaos_types_and_names'), (SELECT id FROM dataset WHERE name='chaos_types_and_names'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='do_bit'), (SELECT id FROM dataset WHERE name='do_bit'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='edns_version'), (SELECT id FROM dataset WHERE name='edns_version'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='idn_qname'), (SELECT id FROM dataset WHERE name='idn_qname'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='opcode'), (SELECT id FROM dataset WHERE name='opcode'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rd_bit'), (SELECT id FROM dataset WHERE name='rd_bit'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='qtype'), (SELECT id FROM dataset WHERE name='qtype'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='dnssec_qtype'), (SELECT id FROM dataset WHERE name='dnssec_qtype'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='certain_qnames_vs_qtype'), (SELECT id FROM dataset WHERE name='certain_qnames_vs_qtype'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='qtype_vs_tld'), (SELECT id FROM dataset WHERE name='qtype_vs_tld'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='qtype_vs_qnamelen'), (SELECT id FROM dataset WHERE name='qtype_vs_qnamelen'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode'), (SELECT id FROM dataset WHERE name='rcode'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode_vs_replylen'), (SELECT id FROM dataset WHERE name='rcode_vs_replylen'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode_vs_replylen_big'), (SELECT id FROM dataset WHERE name='rcode_vs_replylen'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='dns_ip_version'), (SELECT id FROM dataset WHERE name='dns_ip_version'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='dns_ip_version_vs_qtype'), (SELECT id FROM dataset WHERE name='dns_ip_version_vs_qtype'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='direction_vs_ipproto'), (SELECT id FROM dataset WHERE name='direction_vs_ipproto'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='transport_vs_qtype'), (SELECT id FROM dataset WHERE name='transport_vs_qtype'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='pcap_stats'), (SELECT id FROM dataset WHERE name='pcap_stats'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet_accum'), (SELECT id FROM dataset WHERE name='client_subnet_accum'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_addr_vs_rcode_accum'), (SELECT id FROM dataset WHERE name='client_addr_vs_rcode_accum'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='ipv6_rsn_abusers_accum'), (SELECT id FROM dataset WHERE name='ipv6_rsn_abusers_accum'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet2_accum'), (SELECT id FROM dataset WHERE name='client_subnet2_accum'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet2_count'), (SELECT id FROM dataset WHERE name='client_subnet2_count'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet2_trace'), (SELECT id FROM dataset WHERE name='client_subnet2_trace'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode_volume'), (SELECT id FROM dataset WHERE name='rcode'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_small'), (SELECT id FROM dataset WHERE name='traffic_sizes_queries'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_small'), (SELECT id FROM dataset WHERE name='traffic_sizes_responses'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_big'), (SELECT id FROM dataset WHERE name='traffic_sizes_queries'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_big'), (SELECT id FROM dataset WHERE name='traffic_sizes_responses'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_volume'), (SELECT id FROM dataset WHERE name='traffic_volume_queries'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_volume'), (SELECT id FROM dataset WHERE name='traffic_volume_responses'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='unique_sources'), (SELECT id FROM dataset WHERE name='unique_sources'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id FROM visible_plot WHERE name='traffic_volume_difference'), (SELECT id FROM dataset WHERE name='traffic_volume_queries'));
+INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id FROM visible_plot WHERE name='traffic_volume_difference'), (SELECT id FROM dataset WHERE name='traffic_volume_responses'));
+
+
+INSERT INTO query_classification (name, title) VALUES ('Malformed', 'The DNS message was malformed and could not be entirely parsed');
+INSERT INTO query_classification (name, title) VALUES ('Src port 0', 'The UDP query came from source port 0');
+INSERT INTO query_classification (name, title) VALUES ('Funny Qtype', 'Query type was not one of the documented types');
+INSERT INTO query_classification (name, title) VALUES ('Funny Qclass', 'Query class was not IN');
+INSERT INTO query_classification (name, title) VALUES ('RFC1918PTR', 'The query type was PTR and the name was in an in-addr.arpa zone covered by RFC1918 private address space');
+INSERT INTO query_classification (name, title) VALUES ('A-for-.', 'The query name was empty (equal to the root zone)');
+INSERT INTO query_classification (name, title) VALUES ('A-for-A', 'The query name was already an IPv4 address');
+INSERT INTO query_classification (name, title) VALUES ('localhost', 'The query was for localhost');
+INSERT INTO query_classification (name, title) VALUES ('root-servers.net', 'The query was for a root-servers.net name');
+INSERT INTO query_classification (name, title) VALUES ('Non-Authoritative TLD', ' The query was for a known-invalid TLD');
+INSERT INTO query_classification (name, title) VALUES ('Unclassified', 'the query did not fall INTO one of the other categories');
+
+
+
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values
+    ('IP', 1, 'ICMP', 'Internet Control Message'),
+    ('IP', 6, 'TCP', 'Transmission Control'),
+    ('IP', 17, 'UDP', 'User Datagram'),
+    ('IP', 58, 'IPv6-ICMP', 'ICMP for IPv6');
+
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values
+    ('DNS CLASS', 0, 'Reserved', 'Reserved'),
+    ('DNS CLASS', 1, 'IN', 'Internet'),
+    ('DNS CLASS', 2, 'Unassigned', 'Unassigned'),
+    ('DNS CLASS', 3, 'CH', 'Chaos'),
+    ('DNS CLASS', 4, 'HS', 'Hesiod'),
+    ('DNS CLASS', 254, 'QCLASS NONE', 'QCLASS NONE'),
+    ('DNS CLASS', 4, 'ANY', 'QCLASS *'),
+    ('DNS CLASS', 65535, 'Reserved', 'Reserved');
+
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values
+    ('qtype', 1, 'A', 'a host address'),
+    ('qtype', 2, 'NS', 'an authoritative name server'),
+    ('qtype', 3, 'MD', 'a mail destination (OBSOLETE - use MX)'),
+    ('qtype', 4, 'MF', 'a mail forwarder (OBSOLETE - use MX)'),
+    ('qtype', 5, 'CNAME', 'the canonical name for an alias'),
+    ('qtype', 6, 'SOA', 'marks the start of a zone of authority'),
+    ('qtype', 7, 'MB', 'a mailbox domain name (EXPERIMENTAL)'),
+    ('qtype', 8, 'MG', 'a mail group member (EXPERIMENTAL)'),
+    ('qtype', 9, 'MR', 'a mail rename domain name (EXPERIMENTAL)'),
+    ('qtype', 10, 'NULL', 'a null RR (EXPERIMENTAL)'),
+    ('qtype', 11, 'WKS', 'a well known service description'),
+    ('qtype', 12, 'PTR', 'a domain name pointer'),
+    ('qtype', 13, 'HINFO', 'host information'),
+    ('qtype', 14, 'MINFO', 'mailbox or mail list information'),
+    ('qtype', 15, 'MX', 'mail exchange'),
+    ('qtype', 16, 'TXT', 'text strings'),
+    ('qtype', 17, 'RP', 'for Responsible Person'),
+    ('qtype', 18, 'AFSDB', 'for AFS Data Base location'),
+    ('qtype', 19, 'X25', 'for X.25 PSDN address'),
+    ('qtype', 20, 'ISDN', 'for ISDN address'),
+    ('qtype', 21, 'RT', 'for Route Through'),
+    ('qtype', 22, 'NSAP', 'for NSAP address, NSAP style A record'),
+    ('qtype', 23, 'NSAP-PTR', 'for domain name pointer, NSAP style'),
+    ('qtype', 24, 'SIG', 'for security signature'),
+    ('qtype', 25, 'KEY', 'for security key'),
+    ('dnssec_qtype', 25, 'KEY', 'for security key'),
+    ('qtype', 26, 'PX', 'fX.400 mail mapping information'),
+    ('qtype', 27, 'GPOS', 'Geographical Position'),
+    ('qtype', 28, 'AAAA', 'IP6 Address'),
+    ('qtype', 29, 'LOC', 'Location Information'),
+    ('qtype', 30, 'NXT', 'Next Domain (OBSOLETE)'),
+    ('qtype', 31, 'EID', 'Endpoint Identifier'),
+    ('qtype', 32, 'NIMLOC', 'Nimrod Locator'),
+    ('qtype', 33, 'SRV', 'Server Selection'),
+    ('qtype', 34, 'ATMA', 'ATM Address'),
+    ('qtype', 35, 'NAPTR', 'Naming Authority Pointer'),
+    ('qtype', 36, 'KX', 'Key Exchanger'),
+    ('qtype', 37, 'CERT', 'CERT'),
+    ('qtype', 38, 'A6', 'A6 (OBSOLETE - use AAAA)'),
+    ('qtype', 39, 'DNAME', 'DNAME'),
+    ('qtype', 40, 'SINK', 'SINK'),
+    ('qtype', 41, 'OPT', 'OPT'),
+    ('qtype', 42, 'APL', 'APL'),
+    ('qtype', 43, 'DS', 'Delegation Signer'),
+    ('dnssec_qtype', 43, 'DS', 'Delegation Signer'),
+    ('qtype', 44, 'SSHFP', 'SSH Key Fingerprint'),
+    ('qtype', 45, 'IPSECKEY', 'IPSECKEY'),
+    ('qtype', 46, 'RRSIG', 'RRSIG'),
+    ('dnssec_qtype', 46, 'RRSIG', 'RRSIG'),
+    ('qtype', 47, 'NSEC', 'NSEC'),
+    ('dnssec_qtype', 47, 'NSEC', 'NSEC'),
+    ('qtype', 48, 'DNSKEY', 'DNSKEY'),
+    ('dnssec_qtype', 48, 'DNSKEY', 'DNSKEY'),
+    ('qtype', 49, 'DHCID', 'DHCID'),
+    ('qtype', 50, 'NSEC3', 'NSEC3'),
+    ('dnssec_qtype', 50, 'NSEC3', 'NSEC3'),
+    ('qtype', 51, 'NSEC3PARAM', 'NSEC3PARAM'),
+    ('dnssec_qtype', 51, 'NSEC3PARAM', 'NSEC3PARAM'),
+    ('qtype', 52, 'TLSA', 'TLSA'),
+    ('qtype', 55, 'HIP', 'Host Identity Protocol'),
+    ('qtype', 56, 'NINFO', 'NINFO'),
+    ('qtype', 57, 'RKEY', 'RKEY'),
+    ('qtype', 58, 'TALINK', 'Trust Anchor LINK'),
+    ('qtype', 59, 'CDS', 'Child DS'),
+    ('qtype', 99, 'SPF', NULL),
+    ('qtype', 100, 'UINFO', NULL),
+    ('qtype', 101, 'UID', NULL),
+    ('qtype', 102, 'GID', NULL),
+    ('qtype', 103, 'UNSPEC', NULL),
+    ('qtype', 104, 'NID', NULL),
+    ('qtype', 105, 'L32', NULL),
+    ('qtype', 106, 'L64', NULL),
+    ('qtype', 107, 'LP', NULL),
+    ('qtype', 249, 'TKEY', 'Transaction Key'),
+    ('qtype', 250, 'TSIG', 'Transaction Signature'),
+    ('qtype', 251, 'IXFR', 'incremental transfer'),
+    ('qtype', 252, 'AXFR', 'transfer of an entire zone'),
+    ('qtype', 253, 'MAILB', 'mailbox-related RRs (MB, MG or MR)'),
+    ('qtype', 254, 'MAILA', 'mail agent RRs (OBSOLETE - see MX)'),
+    ('qtype', 255, 'ANY', 'A request for all records'),
+    ('qtype', 256, 'URI', 'URI'),
+    ('qtype', 257, 'CAA', 'Certification Authority Authorization'),
+    ('qtype', 32768, 'TA', 'DNSSEC Trust Authorities'),
+    ('qtype', 32769, 'DLV', 'DNSSEC Lookaside Validation'),
+    ('qtype', 65535, 'Reserved', NULL);
+
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values
+    ('opcode', 0, 'Query', NULL),
+    ('opcode', 1, 'IQuery', '(Inverse Query, OBSOLETE)'),
+    ('opcode', 2, 'Status', NULL),
+    ('opcode', 3, 'Unassigned', NULL),
+    ('opcode', 4, 'Notify', NULL),
+    ('opcode', 5, 'Update', NULL);
+
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values
+    ('rcode', 0, 'NoError', 'No Error'),
+    ('rcode', 1, 'FormErr', 'Format Error'),
+    ('rcode', 2, 'ServFail', 'Server Failure'),
+    ('rcode', 3, 'NXDomain', 'Non-Existent Domain'),
+    ('rcode', 4, 'NotImp', 'Not Implemented'),
+    ('rcode', 5, 'Refused', 'Query Refused'),
+    ('rcode', 6, 'YXDomain', 'Name Exists when it should not'),
+    ('rcode', 7, 'YXRRSet', 'RR Set Exists when it should not'),
+    ('rcode', 8, 'NXRRSet', 'RR Set that should exist does not'),
+    ('rcode', 9, 'NotAuth', 'Server Not Authoritative for zone'),
+    ('rcode', 10, 'NotZone', 'Name not contained in zone'),
+    ('rcode', 16, 'BADVERS', 'Bad OPT Version'),
+    ('rcode', 17, 'BADKEY', 'Key not recognized'),
+    ('rcode', 18, 'BADTIME', 'Signature out of time window'),
+    ('rcode', 19, 'BADMODE', 'Bad TKEY Mode'),
+    ('rcode', 20, 'BADNAME', 'Duplicate key name'),
+    ('rcode', 21, 'BADALG', 'Algorithm not supported'),
+    ('rcode', 22, 'BADTRUNC', 'Bad Truncation'),
+    ('rcode', 65535, 'Reserved', 'Reserved, can be allocated by Standards Action');
+
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values
+    ('DNS EDNS0 Options', 0, 'Reserved', NULL),
+    ('DNS EDNS0 Options', 1, 'LLQ', 'On-hold'),
+    ('DNS EDNS0 Options', 2, 'UL', 'On-hold'),
+    ('DNS EDNS0 Options', 3, 'NSID', 'Standard'),
+    ('DNS EDNS0 Options', 4, 'Reserved', NULL);
+
+-- Fill in the blanks
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'DNS CLASS' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(5,253) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'DNS CLASS' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(256,65279) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'DNS CLASS' AS registry, v AS value, 'Reserved for Private Use' AS name
+FROM generate_series(65280,65534) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(53,54) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(60,98) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(108,248) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(258,32767) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(32770,65279) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'qtype' AS registry, v AS value, 'Reserved for Private Use' AS name
+FROM generate_series(65280,65534) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'opcode' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(6,15) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'rcode' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(11,15) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'rcode' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(23,3840) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'rcode' AS registry, v AS value, 'Reserved for Private Use' AS name
+FROM generate_series(3841,4095) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'rcode' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(4096,65534) as v
+);
+
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'DNS EDNS0 Options' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(5,65534) as v
+);
+
+-- These are flags - not sure where I am going to put them.
+/*
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values
+    ('DNS Header Flags', 5, 'AA', 'Authoritative Answer'),
+    ('DNS Header Flags', 6, 'TC', 'Truncated Response'),
+    ('DNS Header Flags', 7, 'RD', 'Recursion Desired'),
+    ('DNS Header Flags', 8, 'RA', 'Recursion Allowed'),
+    ('DNS Header Flags', 9, 'Reserved', 'Reserved'),
+    ('DNS Header Flags', 10, 'AD', 'Authentic Data'),
+    ('DNS Header Flags', 11, 'CD', 'Checking Disabled');
+
+INSERT INTO dsc.iana_lookup (registry,value,name,description) values ('EDNS Header Flags', 0, 'DO', 'DNSSEC answer OK');
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'EDNS Header Flags' AS registry, v AS value, 'Reserved' AS name
+FROM generate_series(5,65534) as v
+);
+
+ INSERT INTO dsc.iana_lookup (registry,value,name,description) values ('EDNS version Number', 0, 'EDNS version 0', 'EDNS version 0');
+INSERT INTO dsc.iana_lookup
+(
+SELECT 'EDNS version Number' AS registry, v AS value, 'Unassigned' AS name
+FROM generate_series(5,65534) as v
+);
+*/
diff --git a/tools/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in b/tools/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in
new file mode 100755
index 0000000..e4efdcc
--- /dev/null
+++ b/tools/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in
@@ -0,0 +1,37 @@
+#!@PYTHON_EXE@
+# 
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+import csv
+with open('/tmp/GeoLite2-City-Blocks-IPv4.csv', 'rb') as f:
+	reader = csv.reader(f)
+	with open('/tmp/blocks_ready_to_import.csv', 'wb') as o:
+		writer = csv.writer(o)
+		for row in reader:
+			if row[0] == "network":
+				continue
+			if row[1] == "":
+				writer.writerow([row[0],row[2],row[7],row[8]])
+			else:
+				writer.writerow([row[0],row[1],row[7],row[8]])
+	o.close()
+f.close()
+with open('/tmp/GeoLite2-City-Blocks-IPv6.csv', 'rb') as f:
+	reader = csv.reader(f)
+	with open('/tmp/blocks_ready_to_import.csv', 'ab') as o:
+		writer = csv.writer(o)
+		for row in reader:
+			if row[0] == "network":
+				continue
+			if row[1] == "":
+				writer.writerow([row[0],row[2],row[7],row[8]])
+			else:
+				writer.writerow([row[0],row[1],row[7],row[8]])
+	o.close()
+f.close()
\ No newline at end of file
diff --git a/tools/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in b/tools/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in
new file mode 100755
index 0000000..947ed2f
--- /dev/null
+++ b/tools/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in
@@ -0,0 +1,21 @@
+#!@PYTHON_EXE@
+# 
+# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+import csv
+with open('/tmp/GeoLite2-City-Locations-en.csv', 'rb') as f:
+	reader = csv.reader(f)
+	with open('/tmp/locations_ready_to_import.csv', 'ab') as o:
+		writer = csv.writer(o)
+		for row in reader:
+			if row[0] == "geoname_id":
+				continue
+			writer.writerow([row[0],row[5],row[10]])
+	o.close()
+f.close()
\ No newline at end of file
diff --git a/tools/hedgehogctl.in b/tools/hedgehogctl.in
new file mode 100755
index 0000000..0197b38
--- /dev/null
+++ b/tools/hedgehogctl.in
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+# Wrapper script to run the other tools
+# The other tools will be installed in libexecdir
+
+set -e
+# set -x 
+
+ALL_TOOLS=( database_init \
+        database_manage_partitions \
+        database_process_rssac_data \
+        database_rm_old_rssac_data \
+        database_update_geoip \
+        database_update_nodes \
+        database_update_service_addr \
+        database_update_tlds \
+        database_update_tlds_from_zone \
+        datafiles_create_summary \
+        datafiles_rm_empty_xml_dirs \
+        datafiles_tar_old_xml_files \
+        plotcache_generate_cached_plots \
+        plotcache_rm_cached_plots \
+        rssac_generate_reports )
+
+SCRIPT_DIR=@LIBEXEC@
+HELP=0
+FIRST_COMMAND_ARG=2
+COMMAND=${1}
+shopt -s nocasematch
+
+
+usage () {
+	echo
+	echo "Run a Hedgehog command"
+	echo
+	echo "Usage: $(basename $0) [help|list] [COMMAND] [command_options]"
+	echo
+  if [ -z "$1" ] ; then
+	  echo "The list of COMMANDs is:"
+  	printf '  %s\n' "${TOOLS[@]}"
+  else
+    echo "The list of available COMMANDs can be found by using the list option."
+  fi
+	echo
+  exit 1
+}
+
+if [[  " ${1} " == " help2man " ]] ; then
+  usage help2man
+fi
+
+# Create new list from just commands that are installed
+for cmd in ${ALL_TOOLS[@]} ; do
+  if [ -x ${SCRIPT_DIR}/${cmd} ] ; then
+    TOOLS=("${TOOLS[@]}" $cmd)
+  fi
+done
+
+if [[  " ${1} " == " list " ]] ; then
+  echo "Valid commands are:"
+  printf '  %s\n' "${TOOLS[@]}"
+  echo
+  exit 0
+fi
+
+if [[  " ${1} " == " help " ]] ; then
+  COMMAND=${2}
+  HELP=1
+  if [ -z ${COMMAND} ] ; then
+    usage
+  fi
+fi
+
+if [[ ! " ${TOOLS[*]} " =~ " ${COMMAND} " ]] ; then
+  echo
+  echo "Invalid command: ${COMMAND}"
+  usage
+fi
+
+if [[ ${HELP} -eq 1 ]] ; then
+  ${SCRIPT_DIR}/${COMMAND} -h
+  exit 0
+fi
+
+exec ${SCRIPT_DIR}/${COMMAND} ${@:$FIRST_COMMAND_ARG} 
\ No newline at end of file
diff --git a/tools/refile_and_grok.in b/tools/refile_and_grok.in
new file mode 100755
index 0000000..433544c
--- /dev/null
+++ b/tools/refile_and_grok.in
@@ -0,0 +1,177 @@
+#!/bin/bash
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Developed by Sinodun IT (www.sinodun.com)
+
+#
+# File: refile_and_grok  
+#
+
+PROG=$(basename "$0" )
+
+# We need a write user with access to the data dir
+a=$(@LIBEXEC@/check_user_perms -wfc)
+if [ $? -ne 0 ] ; then
+	echo $a
+	exit 1
+fi
+eval $a
+export PGPASSWORD=$PGPASSWORD
+
+set -e # exit if any command yields a non 0 exit status
+WD="$DATADIR"
+INPUT="XML"
+OUTPUT="DB"
+START_DATE=""
+RSSAC_FLAG=""
+RESERVED_CPUS=""
+PROCESSES=""
+ARGS=""
+LOG="dsc-extractor.out"
+APPEND_LOG=0
+NUM_RUNNING=0
+
+usage () {
+    echo "${PROG} - finds all input files in working directory and processes to output format"
+    echo 
+    echo "-w Working directory to search for input files (default: ${WD})"
+    echo
+    echo "-i Input file format <XML|DAT> (default: ${INPUT})"
+    echo
+    echo "-o Output file format <DAT|DB> (default: ${OUTPUT})"
+    echo
+    echo "-c Non-interactive mode - use this flag when being run by a cron job"
+    echo
+    echo "-s Start date from which to process incoming data (XML input only)"
+    echo
+    echo "-r Disable processing of rssac data. Default is to process all data."
+    echo
+    echo "-R Reserved real cores. Number of CPU cores to exclude from import (default 0). Specify either -R or -P."
+    echo
+    echo "-P Number of processes to launch (default 1 per real CPU core). Specify either -R or -P."
+    echo
+    echo "-a Append output to the $PROG.stdout file (default is overwrite)"
+    echo
+    echo "-h Show this help."
+    echo
+}
+
+
+while getopts ":w:i:o:cs:raR:P:h" opt; do
+    case "$opt" in
+        w  ) WD=$OPTARG ;;
+        i  ) INPUT=$OPTARG ;;
+        o  ) OUTPUT=$OPTARG ;;
+        s  ) START_DATE="-s $OPTARG" ;;
+        r  ) RSSAC_FLAG="-r" ;;
+        R  ) RESERVED_CPUS=$OPTARG ;;
+        P  ) PROCESSES=$OPTARG ;;
+        c  ) CRON=1 ;;
+        a  ) APPEND_LOG=1 ;;
+        h  ) usage
+             exit 1 ;;
+        \? ) usage
+             exit 1 ;;
+    esac
+done
+
+if [[ -n "$RESERVED_CPUS"  &&  -n "$PROCESSES" ]] ; then
+	echo "Reserved CPUs ($RESERVED_CPUS) and Number of processes ($PROCESSES) cannot both be specified. Exiting."
+	exit
+fi
+
+if [[ -n "$PROCESSES"  &&  ! "$PROCESSES" =~ ^[0-9]+$ ]] ; then
+	echo "Number of processes ($PROCESSES) is not a positive number. Exiting."
+	exit
+fi
+
+if [[ -n "$RESERVED_CPUS"  && ! "$RESERVED_CPUS" =~ ^[0-9]+$ ]] ; then
+	echo "Reserved CPUS ($RESERVED_CPUS) is not a positive number. Exiting."
+	exit
+fi
+#
+# if [[ -z "$RESERVED_CPUS" ]] ; then
+#     RESERVED_CPUS=0
+# fi
+
+# Get number of real CPUs
+export REAL_CPUS=$(lscpu | awk -F: 'BEGIN { IGNORECASE=1; total=1 } /socket/ { total=total * $2 } END { print total }')
+
+# At this point we have either reserved or used or nothing set
+if [[ -n "$RESERVED_CPUS" ]] ; then
+    if [ "$RESERVED_CPUS" -ge "$REAL_CPUS" ] ; then
+    	echo "Reserved CPUS ($RESERVED_CPUS) is greater than or equal to the number of available real CPUS ($REAL_CPUS). Exiting."
+    	exit
+    fi
+    # This option allows some CPUS to be reserved for the web front end in the case that the import is running continuously
+    CPUS=$(( REAL_CPUS - RESERVED_CPUS ))
+    echo "Using $CPUS processes on $REAL_CPUS real CPUs ($RESERVED_CPUS real CPUs reserved)."
+elif [[ -n "$PROCESSES" ]] ; then
+    CPUS=$PROCESSES
+    echo "Using $CPUS Processes as requested by user."
+else
+    # The user didn't specify anything so fallback to defaults
+    CPUS=$REAL_CPUS
+    echo "Using $CPUS processes on $CPUS real CPUs (no real CPUs reserved)."
+fi
+
+
+# Check to see if dsc-extractor is already running and if it is then only start
+# CPUS - num_running_dsc-extractors
+NUM_RUNNING=$(ps -ef | grep -v grep | awk ' { print $8 } ' | grep dsc-extractor | wc -l)
+CPUS=$(( CPUS - NUM_RUNNING ))
+if [ $CPUS -le 0 ] ; then
+    echo "$(date) Too many dsc-extractor processes already running, exiting"
+    exit 1
+fi
+echo "$(date) Starting $CPUS new processes."
+
+if [[ "$INPUT" = "XML" || "$INPUT" = "DAT" ]] && [ "$OUTPUT" = "DB" ]
+then
+    ARGS="-i ${INPUT} -o PG_DB -c \"${CONN_STRING_XX}\" ${START_DATE} ${RSSAC_FLAG}"
+    LOG=${LOG/.out/-xml-db.out}
+    [ "$INPUT" == "DAT" ] && LOG=${LOG/xml/dat}
+fi
+
+
+if [ -z "$CRON" ]
+then
+    echo "Running ${PROG} ${INPUT}->${OUTPUT} from ${WD}"
+    echo "ARGS = $ARGS"
+    echo "LOG = $LOG"
+    echo -n "Would you like to continue? y/N: "
+    read c
+    [[ ! "$c" =~ ^(Y|y) ]] && exit 1
+fi
+
+#set -x #Print commands just before execution - with all expansions and substitutions done, and words marked - useful for debugging.
+
+cd "${WD}"
+if [ $APPEND_LOG -eq 1 ]; then
+	exec >>"$PROG.stdout"
+else 
+	exec >"$PROG.stdout"
+fi
+#exec 2>&1
+
+echo "$(date) SCRIPT COMMENCED"
+BINDIR=@BIN@
+export BINDIR SERVER
+for SERVER in $(psql $CONN_STRING -tc "select display_name from server;") ; do
+    [ -L "$SERVER" ] && continue
+    [ -d "$SERVER" ] || continue
+    ls "${SERVER}" | xargs -i --max-procs=$CPUS bash -c "[ -L \"${SERVER}/{}\" ] || ([ -d \"${SERVER}/{}\" ] && cd \"${SERVER}/{}\"; echo \"\$(date) START: ${SERVER}/{}\"; $BINDIR/dsc-extractor ${ARGS} >> \"${LOG}\" 2>&1; echo \"\$(date) DONE: ${SERVER}/{}\")"
+    # use below version instead to skip nodes that don't
+    # have an incoming directory.  It is useful if you
+    # receive pre-grokked data to this presenter, but
+    # NOT useful if you need backward compatibility with
+    # older collectors that do not put XMLs into an
+    # incoming subdirectory
+    # ls "${SERVER}" | xargs -i --max-procs=$CPUS bash -c "[ -L \"${SERVER}/{}\" ] || ([ -d \"${SERVER}/{}\" ] && [ -d \"${SERVER}/{}/incoming\" ] && cd \"${SERVER}/{}\"; echo \"`date`-->START: ${SERVER}/{}\"; $BINDIR/dsc-extractor ${ARGS} >> \"${LOG}\" 2>&1; echo \"`date`-->DONE: ${SERVER}/{}\")"
+done
+echo "$(date) SCRIPT COMPLETE"
diff --git a/www/Makefile.am b/www/Makefile.am
new file mode 100755
index 0000000..7f9c177
--- /dev/null
+++ b/www/Makefile.am
@@ -0,0 +1,83 @@
+# TODO: Split this makefile up
+# TODO: misleading error message when trying to remove RPostgreSQLHelper if not there: sudo R CMD REMOVE RPostgreSQLHelper || true
+
+CLEANFILES = $(data_DATA) $(nobase_data_DATA)
+
+sysconfdir = @sysconfdir@/hedgehog
+
+nobase_data_DATA =  hedgehog/conf/hedgehog.conf \
+					hedgehog/conf/hedgehog_webdav.conf \
+					hedgehog/www/R/startup.R \
+					hedgehog/www/hedgehog.html \
+					hedgehog/www/about.html \
+					hedgehog/www/brew/hedgehog.brew 
+
+nobase_dist_data_DATA = hedgehog/www/images/ajax-loader.gif \
+						hedgehog/www/images/apple-touch-icon-114x114.png \
+						hedgehog/www/images/apple-touch-icon.png \
+						hedgehog/www/images/apple-touch-icon-72x72.png \
+						hedgehog/www/images/calendar.png \
+						hedgehog/www/images/favicon.ico \
+						hedgehog/www/images/none.png \
+						hedgehog/www/images/some.png \
+						hedgehog/www/images/all.png \
+						hedgehog/www/images/info.png \
+						hedgehog/www/images/help.png \
+						hedgehog/www/js/datetimes.js \
+						hedgehog/www/js/gengraph.js \
+						hedgehog/www/js/nodetabs.js \
+						hedgehog/www/plots/no_graph.png \
+						hedgehog/www/plots/no_results.png \
+						hedgehog/www/plots/no_connection.png \
+						hedgehog/www/R/generatePlot.R \
+						hedgehog/www/R/prepareStatement.R \
+						hedgehog/www/R/utilities.R \
+						hedgehog/www/stylesheets/base.css \
+						hedgehog/www/stylesheets/layout.css \
+						hedgehog/www/stylesheets/skeleton.css \
+						hedgehog/R_packages/RPostgreSQLHelper/DESCRIPTION \
+						hedgehog/R_packages/RPostgreSQLHelper/man/dbGetDataFrame.Rd \
+						hedgehog/R_packages/RPostgreSQLHelper/man/dbGetResultSet.Rd \
+						hedgehog/R_packages/RPostgreSQLHelper/man/maxAttempts.Rd \
+						hedgehog/R_packages/RPostgreSQLHelper/man/RPostgreSQLHelper-package.Rd \
+						hedgehog/R_packages/RPostgreSQLHelper/NAMESPACE \
+						hedgehog/R_packages/RPostgreSQLHelper/R/dbGetFunctions.r \
+						hedgehog/conf/rssac.yaml
+subst_PACKAGEVERSION = sed -e "s,@HH_PACKAGE_VERSION@,@PACKAGE_VERSION@," -e "s|@HH_VERSION_NAME@|@VERSION_NAME@|"
+subst_DB_SCHEMA_VERSION = sed -e 's|DB_SCHEMA_VERSION|@DB_SCHEMA_VERSION@|'
+
+hedgehog/conf/hedgehog.conf: $(srcdir)/hedgehog/conf/hedgehog.conf.in Makefile
+	mkdir -p hedgehog/conf/
+	$(subst_HH_DIRS) < $(srcdir)/hedgehog/conf/hedgehog.conf.in > hedgehog/conf/hedgehog.conf
+
+hedgehog/conf/Rapache_hedgehog.load: $(srcdir)/hedgehog/conf/Rapache_hedgehog.load.in Makefile
+	mkdir -p hedgehog/conf/
+	$(subst_HH_DIRS) < $(srcdir)/hedgehog/conf/Rapache_hedgehog.load.in > hedgehog/conf/Rapache_hedgehog.load
+
+hedgehog/www/R/startup.R: $(srcdir)/hedgehog/www/R/startup.R.in Makefile
+	mkdir -p hedgehog/www/R/
+	$(subst_HH_CONF_FILE) < $(srcdir)/hedgehog/www/R/startup.R.in > hedgehog/www/R/startup.R	
+
+hedgehog/www/hedgehog.html: $(srcdir)/hedgehog/www/hedgehog.html.in Makefile
+	$(subst_PACKAGEVERSION) < $(srcdir)/hedgehog/www/hedgehog.html.in > hedgehog/www/hedgehog.html
+
+hedgehog/www/about.html: $(srcdir)/hedgehog/www/about.html.in Makefile
+	$(subst_PACKAGEVERSION) < $(srcdir)/hedgehog/www/about.html.in > hedgehog/www/about.html
+
+hedgehog/www/brew/hedgehog.brew: $(srcdir)/hedgehog/www/brew/hedgehog.brew.in Makefile
+	mkdir -p hedgehog/www/brew/
+	$(subst_DB_SCHEMA_VERSION) < $(srcdir)/hedgehog/www/brew/hedgehog.brew.in | $(subst_HH_DIRS)  > hedgehog/www/brew/hedgehog.brew
+
+EXTRA_DIST = 	$(srcdir)/hedgehog/www/hedgehog.html.in \
+				$(srcdir)/hedgehog/www/about.html.in \
+				$(srcdir)/hedgehog/www/R/startup.R.in \
+				$(srcdir)/hedgehog/conf/hedgehog.conf.in \
+				$(srcdir)/hedgehog/conf/Rapache_hedgehog.load.in \
+				$(srcdir)/hedgehog/www/brew/hedgehog.brew.in
+
+install-data-hook:
+	mkdir -p ${DESTDIR}$(localstatedir)/hedgehog/www
+	chmod -R 775 ${DESTDIR}$(localstatedir)/hedgehog/www
+	test -f ${DESTDIR}${sysconfdir}/hedgehog_gui.yaml || \
+		${INSTALL_DATA} -m 644 $(srcdir)/$(hh_web_conf_dir)/hedgehog_gui.yaml ${DESTDIR}${sysconfdir}
+		${INSTALL_DATA} -m 644 $(srcdir)/$(hh_web_conf_dir)/hedgehog_gui.yaml ${DESTDIR}${sysconfdir}/hedgehog_gui.yaml.sample
diff --git a/www/hedgehog/R_packages/RPostgreSQLHelper/DESCRIPTION b/www/hedgehog/R_packages/RPostgreSQLHelper/DESCRIPTION
new file mode 100755
index 0000000..c9efde8
--- /dev/null
+++ b/www/hedgehog/R_packages/RPostgreSQLHelper/DESCRIPTION
@@ -0,0 +1,15 @@
+Package: RPostgreSQLHelper
+Title: wrapper functions for the RPostgreSQL package to handle db
+        connection failures, where connections are initialised via
+        rapache config file.
+Version: 1.0
+Depends: RPostgreSQL
+Date: 2013-03-20
+Author: John Dickinson <jad@sinodun.com>
+Maintainer: John Dickinson <jad@sinodun.com>
+Description: Attempts to reconnect to the database if the
+	connection has been lost and fails gracefully in
+        the event that the connection to the database
+        cannot be re-established.
+License: GPL-3
+Packaged: 2013-03-20 16:11:46 UTC; jad
diff --git a/www/hedgehog/R_packages/RPostgreSQLHelper/NAMESPACE b/www/hedgehog/R_packages/RPostgreSQLHelper/NAMESPACE
new file mode 100644
index 0000000..0800728
--- /dev/null
+++ b/www/hedgehog/R_packages/RPostgreSQLHelper/NAMESPACE
@@ -0,0 +1,18 @@
+#
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.# 
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+# EXPORTS
+# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+# Export all public methods, that is, those without a preceeding dot
+# in their names.
+exportPattern("^[^\\.]")
\ No newline at end of file
diff --git a/www/hedgehog/R_packages/RPostgreSQLHelper/R/dbGetFunctions.r b/www/hedgehog/R_packages/RPostgreSQLHelper/R/dbGetFunctions.r
new file mode 100755
index 0000000..b948e00
--- /dev/null
+++ b/www/hedgehog/R_packages/RPostgreSQLHelper/R/dbGetFunctions.r
@@ -0,0 +1,96 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+#   wrapper functions for RPostgreSQL dbSendQuery and
+#   fetch to handle db connection failures, where
+#   connections are initialised via rapache config file.
+#   
+#   ensures that dbSendQuery first attempts to
+#   reconnect to the database if the connection has been
+#   lost (apache2 force-reload) and fails gracefully in
+#   the event that the connection to the database cannot
+#   be re-established.
+#   
+#   dbGetResultSet() will return a result set if successful
+#   or NULL otherwise.
+#   
+#   dbGetDataFrame() will return a data frame if successful
+#   or NULL otherwise.
+#   
+#   dependencies:   RPostgreSQL
+#   
+#   author: John Dickinson <jad@sinodun.com>
+
+
+library("RPostgreSQL")
+
+# max number of attempts to reconnect to the database
+maxAttempts <- 3
+
+
+#   attempt to re-establish a db connection
+.reload <- function(dbiDriver, dbiConnection, dbConnStr){
+    # close any pending result sets
+    resSets = try(dbListResults(dbiConnection))
+    if(class(resSets) != "try-error"){
+        for(resultSet in resSets){
+            dbClearResult(resultSet)
+        }
+    }
+
+    # close all connections that the dbiDriver is managing
+    conns = try(dbListConnections(dbiDriver))
+    if(class(conns) != "try-error"){
+        for(connection in conns){
+            dbDisconnect(connection)
+        }
+    }
+
+    dbiConnection <- eval(parse(text = paste("dbConnect(dbdrv,",  dbConnStr,")")))
+
+}
+
+#   return the result set associated with sqlQueryStr
+#   from dbiConnection. If the dbiConnection has been lost,
+#   attempt to reconnect to the db. If dbiConnection cannot
+#   be established, return NULL
+dbGetResultSet <- function(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr){
+    rs <- NULL
+    for(attempt in 1:(maxAttempts+1)){
+        rs <- try(dbSendQuery(dbiConnection ,sqlQueryStr))
+        if(class(rs) == "try-error"){
+            if(attempt == 1){
+                system('logger -p user.notice Hedgehog: Error sending query to database')
+            }
+            if(attempt == (maxAttempts+1)){
+                system('logger -p user.notice Hedgehog: Critical Giving up trying to send query to the database: administrator action required')
+                rs <- NULL
+            }else{
+                system(paste("logger -p user.notice Hedgehog: Info re-connecting to the database: attempt ", attempt, " of ", maxAttempts, sep=""))
+                .reload(dbiDriver, dbiConnection, dbConnStr)
+            }
+        }else break
+    }
+    return(rs)
+}
+
+#   return the data frame associated with sqlQueryStr
+#   from dbiConnection. 
+dbGetDataFrame <- function(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr){
+    df <- NULL
+    rs <- dbGetResultSet(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
+    if(!is.null(rs)){
+        df <- fetch(rs, n=-1)
+    dbClearResult(rs)
+    }
+    return(df)
+}
\ No newline at end of file
diff --git a/www/hedgehog/R_packages/RPostgreSQLHelper/man/RPostgreSQLHelper-package.Rd b/www/hedgehog/R_packages/RPostgreSQLHelper/man/RPostgreSQLHelper-package.Rd
new file mode 100755
index 0000000..751d835
--- /dev/null
+++ b/www/hedgehog/R_packages/RPostgreSQLHelper/man/RPostgreSQLHelper-package.Rd
@@ -0,0 +1,44 @@
+%%
+%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+%%
+%% This Source Code Form is subject to the terms of the Mozilla Public
+%% License, v. 2.0. If a copy of the MPL was not distributed with this
+%% file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+%%
+%% Developed by Sinodun IT (www.sinodun.com)
+%%
+
+\name{RPostgreSQLHelper-package}
+\alias{RPostgreSQLHelper-package}
+\alias{RPostgreSQLHelper}
+\docType{package}
+\title{RPostgreSQL wrapper functions}
+\description{Handles dbiConnection failures}
+\details{
+\tabular{ll}{
+Package: \tab RPostgreSQLHelper\cr
+Type: \tab Package\cr
+Version: \tab 1.0\cr
+Date: \tab 2013-03-20\cr
+License: \tab GPL-3\cr
+}
+dbSendQuery: dbGetResultSet(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
+
+dbSendQuery + fetch: dbGetDataFrame(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
+}
+\author{John Dickinson <jad@sinodun.com>}
+\references{
+	https://code.google.com/p/rpostgresql/
+	http://stat.bell-labs.com/RS-DBI/doc/html/index.html
+}
+%%~~ Optionally other standard keywords, one per line, from file KEYWORDS in ~~
+%%~~ the R documentation directory ~~
+\keyword{ package }
+\seealso{
+%%~~ Optional links to other man pages, e.g. ~~
+%%~~ \code{\link[<pkg>:<pkg>-package]{<pkg>}} ~~
+}
+\examples{
+%%~~ simple examples of the most important functions ~~
+}
diff --git a/www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetDataFrame.Rd b/www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetDataFrame.Rd
new file mode 100755
index 0000000..523a4b1
--- /dev/null
+++ b/www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetDataFrame.Rd
@@ -0,0 +1,57 @@
+%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+%%
+%% This Source Code Form is subject to the terms of the Mozilla Public
+%% License, v. 2.0. If a copy of the MPL was not distributed with this
+%% file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+%%
+%% Developed by Sinodun IT (www.sinodun.com)
+%%
+
+\name{dbGetDataFrame}
+\alias{dbGetDataFrame}
+%- Also NEED an '\alias' for EACH other topic documented here.
+\title{RPostgreSQL dbSendQuery + fetch wrapper function}
+\description{Handles dbiConnection failures}
+\usage{
+dbGetDataFrame(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
+}
+%- maybe also 'usage' for other objects documented here.
+\arguments{
+  \item{dbiDriver}{
+Database driver dbiDriver
+}
+  \item{dbiConnection}{
+Database connection dbiConnection
+}
+  \item{dbConnStr}{
+Database connection string dbConnStr
+}
+  \item{sqlQueryStr}{
+String containing SQL query to be run
+}
+}
+\details{
+%%  ~~ If necessary, more details than the description above ~~
+}
+\value{if successful, return a DBIResult else returns NULL}
+\references{
+	https://code.google.com/p/rpostgresql/
+	http://stat.bell-labs.com/RS-DBI/doc/html/index.html
+}
+\author{John Dickinson <jad@sinodun.com>}
+\note{
+%%  ~~further notes~~
+}
+
+%% ~Make other sections like Warning with \section{Warning }{....} ~
+
+\seealso{
+%% ~~objects to See Also as \code{\link{help}}, ~~~
+}
+\examples{
+}
+% Add one or more standard keywords, see file 'KEYWORDS' in the
+% R documentation directory.
+\keyword{ ~kwd1 }
+\keyword{ ~kwd2 }% __ONLY ONE__ keyword per line
diff --git a/www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetResultSet.Rd b/www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetResultSet.Rd
new file mode 100755
index 0000000..5228067
--- /dev/null
+++ b/www/hedgehog/R_packages/RPostgreSQLHelper/man/dbGetResultSet.Rd
@@ -0,0 +1,57 @@
+%%
+%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+%%
+%% This Source Code Form is subject to the terms of the Mozilla Public
+%% License, v. 2.0. If a copy of the MPL was not distributed with this
+%% file, you can obtain one at https://mozilla.org/MPL/2.0/.
+%%
+%% Developed by Sinodun IT (www.sinodun.com)
+%%
+
+\name{dbGetResultSet}
+\alias{dbGetResultSet}
+%- Also NEED an '\alias' for EACH other topic documented here.
+\title{RPostgreSQL dbSendQuery wrapper function}
+\description{Handles dbiConnection failures}
+\usage{
+dbGetResultSet(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
+}
+%- maybe also 'usage' for other objects documented here.
+\arguments{
+  \item{dbiDriver}{
+Database driver dbiDriver
+}
+  \item{dbiConnection}{
+Database connection dbiConnection
+}
+  \item{dbConnStr}{
+Database connection string dbConnStr
+}
+  \item{sqlQueryStr}{
+String containing SQL query to be run
+}
+}
+\details{
+%%  ~~ If necessary, more details than the description above ~~
+}
+\value{if successful, return a DBIResult else returns NULL}
+\references{
+	https://code.google.com/p/rpostgresql/
+	http://stat.bell-labs.com/RS-DBI/doc/html/index.html
+}
+\author{John Dickinson <jad@sinodun.com>}
+\note{
+%%  ~~further notes~~
+}
+
+%% ~Make other sections like Warning with \section{Warning }{....} ~
+
+\seealso{
+%% ~~objects to See Also as \code{\link{help}}, ~~~
+}
+\examples{
+}
+% Add one or more standard keywords, see file 'KEYWORDS' in the
+% R documentation directory.
+\keyword{ ~kwd1 }
+\keyword{ ~kwd2 }% __ONLY ONE__ keyword per line
diff --git a/www/hedgehog/R_packages/RPostgreSQLHelper/man/maxAttempts.Rd b/www/hedgehog/R_packages/RPostgreSQLHelper/man/maxAttempts.Rd
new file mode 100755
index 0000000..99f1d2b
--- /dev/null
+++ b/www/hedgehog/R_packages/RPostgreSQLHelper/man/maxAttempts.Rd
@@ -0,0 +1,34 @@
+%%
+%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+%%
+%% This Source Code Form is subject to the terms of the Mozilla Public
+%% License, v. 2.0. If a copy of the MPL was not distributed with this
+%% file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+%%
+%% Developed by Sinodun IT (www.sinodun.com)
+%%
+
+\name{maxAttempts}
+\alias{maxAttempts}
+\docType{data}
+\title{maximum number of attempts to re-connect to the database}
+\description{integer value indicating the maximum number of attempts to re-connect to the database}
+\usage{}
+\format{
+  The format is:
+ num 3
+}
+\details{
+%%  ~~ If necessary, more details than the __description__ above ~~
+}
+\source{
+%%  ~~ reference to a publication or URL from which the data were obtained ~~
+}
+\references{
+%%  ~~ possibly secondary sources and usages ~~
+}
+\examples{
+data(maxAttempts)
+}
+\keyword{datasets}
diff --git a/www/hedgehog/conf/hedgehog.conf.in b/www/hedgehog/conf/hedgehog.conf.in
new file mode 100644
index 0000000..156d6a9
--- /dev/null
+++ b/www/hedgehog/conf/hedgehog.conf.in
@@ -0,0 +1,106 @@
+# 
+## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+##
+## This Source Code Form is subject to the terms of the Mozilla Public
+## License, v. 2.0. If a copy of the MPL was not distributed with this
+## file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+## Developed by Sinodun IT (www.sinodun.com)
+
+<VirtualHost *:80>
+
+	#ServerName www.example.com
+	ServerAdmin webmaster@localhost
+  
+	Alias /hedgehog @HH_WEB_SOURCE@
+	<Directory @HH_WEB_SOURCE@>
+	    RSourceOnStartup @HH_WEB_SOURCE@/R/startup.R
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    DefaultType text/html
+	    Require all granted
+	    DirectoryIndex hedgehog.html
+	</Directory>
+
+	<Directory @HH_WEB_SOURCE@/brew>
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    SetHandler r-script
+	    Require all granted
+	    RHandler brew::brew
+	</Directory>
+
+	<Directory @HH_WEB_SOURCE@/images>
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    Require all granted
+	    SetHandler default-handler
+	</Directory>
+
+	<Directory @HH_WEB_SOURCE@/plots>
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    Require all granted
+	    SetHandler default-handler
+	</Directory>
+
+	<Directory @HH_WEB_SOURCE@/js>
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    Require all granted
+	    SetHandler default-handler
+	</Directory>
+
+	<Directory @HH_WEB_SOURCE@/R>
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    Require all granted
+	    SetHandler r-script
+	</Directory>
+
+	<Directory @HH_WEB_SOURCE@/stylesheets>
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    Require all granted
+	    SetHandler default-handler
+	</Directory>
+	
+	Alias /plotcache @HH_PLOTCACHE@
+	<Directory @HH_PLOTCACHE@>
+	    Options None
+	    AllowOverride None
+	    order allow,deny
+	    allow from all
+	    Require all granted
+	    SetHandler default-handler
+	</Directory>
+
+	Alias /rssac @HH_RSSAC@
+	<Directory @HH_RSSAC@>
+	    Options Indexes
+	    AllowOverride None
+	    order allow,deny
+	    Require all granted
+	    allow from all
+	</Directory>	
+
+	# Prints out a nice report about R running within Apache
+	# Do not use this in production by default
+	#<Location /RApacheInfo>
+	#    SetHandler r-info
+	#</Location>
+</VirtualHost>
diff --git a/www/hedgehog/conf/hedgehog_gui.yaml b/www/hedgehog/conf/hedgehog_gui.yaml
new file mode 100755
index 0000000..4c139c4
--- /dev/null
+++ b/www/hedgehog/conf/hedgehog_gui.yaml
@@ -0,0 +1,34 @@
+
+## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+##
+## This Source Code Form is subject to the terms of the Mozilla Public
+## License, v. 2.0. If a copy of the MPL was not distributed with this
+## file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+## Developed by Sinodun IT (www.sinodun.com)
+
+
+--- 
+# YAML config for hedgehog GUI.
+# NOTE: If this file is changed then apache must be restarted for the changes to take effect
+www:
+  default_plot_type             : interactive # 'static'      -> png plots
+                                              # 'interactive' -> googlevis plots
+  default_interactive_plot_type : svg         # 'flash' -> plot requires flash
+                                              # 'svg'   -> plot is SVG/VML and does not require flash
+                                              # (but with svg plots some legends do not wrap properly)
+  default_node_grouping         : instance    # choose from 'none', 'instance', 'city' or 'country'
+  use_plot_caching              : 1           # '1' -> true, use cached plots when possible
+                                              # '0' -> false, never use cached plots
+  caching_delay_in_hours        : 1           # If 'use_plot_caching=1' then only plots with an end time
+                                              # earlier than this number of hours ago are cached.
+                                              # More recent plots are not cached as data may still be
+                                              # being imported
+  presentation_delay_in_hours   : 0           # Number of hours behind now for which the GUI will display data
+  support_url                   :             # configurable target for "Support" external link on Homepage.
+  default_server                :             # Optionally specify the default server to use in the server
+                                              # drop-down (default is first alphabetically)
+  default_plot_name             : by_region   # default plot name (See Hedgehog internal plot name in the user guide)
+  enable_node_selection         : 1           # '1' -> true, Show the node radio button
+                                              # '0' -> false, hide the node radio button
+...
diff --git a/www/hedgehog/conf/hedgehog_webdav.conf b/www/hedgehog/conf/hedgehog_webdav.conf
new file mode 100644
index 0000000..56e0583
--- /dev/null
+++ b/www/hedgehog/conf/hedgehog_webdav.conf
@@ -0,0 +1,24 @@
+# <IfModule mod_ssl.c>
+# <VirtualHost *:443>
+#         ServerName example.com
+#         SSLEngine on
+#         SSLCertificateFile    Server.pem
+#         SSLCertificateKeyFile Server.key
+#         SSLCACertificateFile cacert.pem
+#         SSLVerifyClient require
+#         SSLVerifyDepth  1
+#         CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined
+#         DavLockDB "/somewhere/DAVLock"
+#         Alias /data /var/lib/hedgehog/data
+#         <Directory /var/lib/hedgehog/data>
+#            DAV On
+#            SSLRequireSSL
+#            Options +Indexes
+#            AllowOverride None
+#            order allow,deny
+#            allow from all
+#            DirectoryIndex disable
+#            Require all granted
+#         </Directory>
+# </VirtualHost>
+# </IfModule>
diff --git a/www/hedgehog/conf/rssac.yaml b/www/hedgehog/conf/rssac.yaml
new file mode 100755
index 0000000..027ed1c
--- /dev/null
+++ b/www/hedgehog/conf/rssac.yaml
@@ -0,0 +1,81 @@
+
+## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+##
+## This Source Code Form is subject to the terms of the Mozilla Public
+## License, v. 2.0. If a copy of the MPL was not distributed with this
+## file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+## Developed by Sinodun IT (www.sinodun.com)
+
+
+--- # YAML config for rssac.
+metric:
+  traffic-volume:
+    statistics_type: kv_pair
+    measurement_parameter: Number of queries
+    service: .root-servers.net
+    statistics:
+      - dns-udp-queries-received-ipv4
+      - dns-udp-queries-received-ipv6
+      - dns-tcp-queries-received-ipv4
+      - dns-tcp-queries-received-ipv6
+      - dns-udp-responses-sent-ipv4
+      - dns-udp-responses-sent-ipv6
+      - dns-tcp-responses-sent-ipv4
+      - dns-tcp-responses-sent-ipv6
+  traffic-sizes:
+    statistics_type: buckets
+    measurement_parameter: Query and response size distribution
+    service: .root-servers.net
+    statistics:
+      - name: udp-request-sizes
+        bucket_min: 16
+        bucket_max: 288
+        bucket_step: 16
+      
+      - name: udp-response-sizes
+        bucket_min: 16
+        bucket_max: 4096
+        bucket_step: 16
+        
+      - name: tcp-request-sizes
+        bucket_min: 16
+        bucket_max: 288
+        bucket_step: 16
+        
+      - name: tcp-response-sizes
+        bucket_min: 16
+        bucket_max: 4096
+        bucket_step: 16
+  rcode-volume:
+    statistics_type: count
+    measurement_parameter: RCODE distribution
+    service: .root-servers.net
+    statistics:
+      - name: rcodes
+  unique-sources:
+    statistics_type: kv_pair_with_alias
+    measurement_parameter: Number of sources seen
+    service: .root-servers.net
+    statistics:
+      - name: num-sources-ipv4
+        alias: IPv4
+        
+      - name: num-sources-ipv6
+        alias: IPv6
+        
+      - name: num-sources-ipv6-aggregate
+        alias: IPv6/64
+  load-time:
+    statistics_type: average
+    measurement_parameter: Latency in publishing available data
+    service: .root-servers.net
+    statistics:
+      - name: time
+  zone-size:
+    statistics_type: value
+    measurement_parameter: The size of the overall root zone
+    service: .root-servers.net
+    statistics:
+      - name: size
+...
diff --git a/www/hedgehog/www/R/generatePlot.R b/www/hedgehog/www/R/generatePlot.R
new file mode 100755
index 0000000..afc9791
--- /dev/null
+++ b/www/hedgehog/www/R/generatePlot.R
@@ -0,0 +1,978 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# TODO(refactor): Refactor this code
+
+# Define globals
+
+# Colours
+GREY <- "#999999"
+ORANGE <- "#E69F00"
+SKYBLUE <- "#56B4E9"
+GREEN <- "#009E73"
+YELLOW <- "#F0E442"
+BLUE <- "#0072B2"
+VERMILLION <- "#D55E00"
+PINK <- "#CC79A7"
+BLACK <- "#000000"
+GRIDGREY <- "#BFBFBF"
+DARKERRED <- "#990F0F"
+DARKERORANGEBT <- "#99700F"
+DARKERLIMEGREEN <- "#1F990F"
+DARKERVIOLET <- "#710F99"
+SOFTRED <- "#E67E7E"
+SOFTORANGE <- "#E6C77E"
+SOFTLIMEGREEN <- "#8AE67E"
+SOFTVIOLET <- "#C77EE6"
+MODERATERED <- "#CC5252"
+MODERATEORANGE <- "#CCA852"
+MODERATELIMEGREEN <- "#60CC52"
+MODERATEVIOLET <- "#A852CC"
+DARKRED <- "#B22D2D"
+DARKORANGEBT <- "#B28B2D"
+DARKLIMEGREEN <- "#3CB22D"
+DARKVIOLET <- "#8B2DB2"
+
+# A colour-blind friendly palette:
+ CBPALETTE = c(DARKERRED, DARKERORANGEBT, DARKERLIMEGREEN, DARKERVIOLET,
+               SOFTRED, SOFTORANGE, SOFTLIMEGREEN, SOFTVIOLET,
+               MODERATERED, MODERATEORANGE, MODERATELIMEGREEN, MODERATEVIOLET,
+               DARKRED, DARKORANGEBT, DARKLIMEGREEN, DARKVIOLET)
+
+NCBPALETTE <- length(CBPALETTE)
+
+MINICBPALETTE <- c(DARKERRED, DARKERLIMEGREEN)
+NMINICBPALETTE <- length(MINICBPALETTE)
+
+# width and height of ggplot pngs
+W <- 940
+H <- 600
+
+##################
+
+linePlot <- function(df, f, title, xlabel, ylabel, gvis) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In linePlot')
+  }
+  
+    if(gvis == 1){
+      # Stupid bug in gvis where the name of the numvar column causes issues with the correct name being diplayed in the legend
+      names(df)[names(df)=="y"] <- "yyyyyyyyyyyy"
+
+      if (gui_config$www$default_interactive_plot_type == "svg") {
+          p <- gvisAnnotationChart(df, numvar="yyyyyyyyyyyy", idvar = "key", datevar = "x", #titlevar="serial",
+                                   options=list(
+                                                #displayAnnotations=TRUE, annotationsWidth=10, 
+                                                legendPosition='newRow', height=540, width=910, colors="['#0072B2','#990F0F','#99700F',
+      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#990F0F','#99700F','#1F990F',
+      '#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99',
+      '#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99','#E67E7E',
+      '#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F',
+      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2']"))
+         
+      }else{
+          p <- gvisAnnotatedTimeLine(df, numvar="yyyyyyyyyyyy", idvar = "key", datevar = "x", 
+                                     options=list(legendPosition='newRow', height=540, width=910, colors="['#0072B2','#990F0F','#99700F',
+      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#990F0F','#99700F','#1F990F',
+      '#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99',
+      '#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99','#E67E7E',
+      '#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F',
+      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2']"))         
+      }
+      title <- sub("\n", "<br />", title)
+      ylabel <- gsub(" +", "&nbsp;", ylabel)
+      p$html$chart['divChart'] <- paste("<div style=\"text-align: center; width: 918px; font-family: HelveticaNeue, 
+                                                      'Helvetica Neue', Helvetica, Arial, sans-serif;\">", 
+                                                       title,
+                                        "</div>",
+                                        "<div style=\"display: flex; display: -webkit-flex; justify-content:flex-start; width: 930px;\">",  
+                                        "<div>", p$html$chart['divChart'], "</div>",
+                                        "<div style=\"width: 17px; height: 17px; font-size: small; transform: translateY(200px) rotate(90deg); transform-origin: right top; 
+                                              -webkit-transform: translateY(200px) rotate(90deg); -webkit-transform-origin: right top;\">", ylabel, "</div>",
+                                        "</div>",
+                                        "<div style=\"text-align: center; width: 918px; font-size: small;\">", xlabel, "</div>",
+                                       sep="")
+      cat(p$html$chart, file=f)
+    }else{
+      nKeys <- length(unique(df$key))
+      df$x <- as.POSIXct(df$x)
+      png(f, width = W, height = H)
+      p <- ggplot(data=df, aes(x=x, y=y, group=key, colour=key)) +
+                  geom_line() +
+                  labs(title=title, x=xlabel, y=ylabel) +
+                  scale_x_datetime(expand=c(0.01,0)) +
+                  scale_y_continuous(expand=c(0.01,0), labels = comma) +
+                  theme_bw() +
+                  theme(panel.grid.major = element_line(colour = GRIDGREY), 
+                        panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
+                  guides(col = guide_legend(nrow = 20, byrow = TRUE, override.aes = list(size=3)))
+      if (nKeys <= NCBPALETTE) {
+        p <- p + scale_colour_manual(values=CBPALETTE)
+      }
+
+      print(p)
+      dev.off()
+
+      fstack <- sub(".png", "-stack.png", f)
+      png(fstack, width = W, height = H)
+      ps <- ggplot(data=df, aes(x=x, y=y, fill=key, order=key)) + 
+                   geom_area(stat="identity") +
+                   labs(title=title, x=xlabel, y=ylabel) +
+                   scale_x_datetime(expand=c(0.01,0)) + 
+                   scale_y_continuous(expand=c(0,0), labels = comma) +
+                   theme_bw() +
+                   theme(panel.grid.major = element_line(colour = GRIDGREY), 
+                         panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
+                   guides(fill = guide_legend(nrow = 20, byrow = TRUE))
+
+      if (nKeys <= NCBPALETTE) {
+        ps <- ps + scale_fill_manual(values=CBPALETTE)
+      }
+      print(ps)
+      dev.off()
+
+      flog <- sub(".png", "-log.png", f)
+      png(flog, width = W, height = H)
+      p <- p + scale_y_log10() + ylab(paste("log(",ylabel,")",sep=""))
+      print(p)
+      dev.off()
+    }
+}
+
+barPlot <- function(df, f, title, xlabel, ylabel, gvis, vertical=0, small=1) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In barPlot')
+  }
+
+    if(gvis == 1){
+      title <- sub("\n", " ", title)
+      if(vertical == 1){
+          p <- gvisColumnChart(df, xvar='x', yvar='y', 
+                            options=list(legend="none", title=title, height=600, width=940,
+                                         vAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'10'}}", sep=""), 
+                                         hAxis=paste("{title:'",xlabel,"',textStyle:{fontSize:'14'}}", sep=""), 
+                                         chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
+      }else{
+        if(small==1){
+          p <- gvisBarChart(df, xvar='x', yvar='y',
+                            options=list(legend="none", title=title, height=600, width=940,
+                                         vAxis=paste("{title:'",xlabel,"',textStyle:{fontSize:'10'}}", sep=""), 
+                                         hAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'14'}}", sep=""), 
+                                         chartArea="{left:100,top:50,width:\"88%\",height:\"80%\"}"))
+        } else {
+          p <- gvisBarChart(df, xvar='x', yvar='y',
+                            options=list(legend="none", title=title, height=600, width=940,
+                                         vAxis=paste("{title:'",xlabel,"'}", sep=""), 
+                                         hAxis=paste("{title:'",ylabel,"', textStyle:{fontSize:'14'}}", sep=""), 
+                                         chartArea="{left:80,top:50,width:\"88%\",height:\"80%\"}"))
+        }
+      }
+      cat(p$html$chart,file=f)
+    }else{
+      if (vertical == 0) {
+          df$x <- factor(df$x, levels=rev(as.character(df$x)))
+      }
+      png(f, width = W, height = H)
+
+      p <- ggplot(data=df, aes(x=x, y=y)) +
+                  geom_bar(fill=GREY, colour=GREY, stat="identity") +
+                  labs(title=title, x=xlabel, y=ylabel) +
+                  theme_bw() + scale_y_continuous(expand=c(0,0), labels = comma) +
+                  theme(panel.grid.major.y = element_line(colour = GRIDGREY), 
+                        panel.grid.minor.y = element_line(colour = GRIDGREY, linetype = "dotted"), 
+                        panel.grid.major.x = element_blank(), panel.grid.minor.x = element_blank())
+
+      if (vertical == 0) {
+        p <- p +
+             coord_flip() +
+             theme(panel.grid.major.x = element_line(colour = GRIDGREY), 
+                   panel.grid.minor.x = element_line(colour = GRIDGREY, linetype = "dotted"), 
+                   panel.grid.major.y = element_blank(), panel.grid.minor.y = element_blank())
+      }
+
+      print(p)
+      dev.off()
+    }
+}
+
+stackedBarPlot <- function(df, f, title, xlabel, ylabel, gvis, pltnm, scalex="discrete", vertical=0, gbar_width=0) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In stackedBarPlot')
+  }
+    if(gvis == 1){
+      title <- sub("\n", " ", title)
+      de <- cast(df, x ~ key, value='y', fun.aggregate=sum)
+      y_var <- tail(colnames(de),-1)
+      if(vertical == 1){
+          if (gbar_width == 0) {
+              p <- gvisColumnChart(de, xvar='x', yvar=y_var,
+                                   options=list(isStacked=TRUE, title=title, height=600, width=940,
+                                                vAxis=paste("{title:'", ylabel,"', format: 'short'}", sep=""), 
+                                                hAxis=paste("{title:'", xlabel,"', textStyle:{fontSize:'14'}}", sep=""), 
+                                                legend=paste("{textStyle:{fontSize:'12'}}", sep=""),
+                                                chartArea="{left:80,top:50,width:\"75%\",height:\"80%\"}"))
+          } else {
+              p <- gvisColumnChart(de, xvar='x', yvar=y_var,
+                                   options=list(isStacked=TRUE, title=title, height=600, width=940,
+                                                vAxis=paste("{title:'", ylabel,"', format: 'short'}", sep=""), 
+                                                hAxis=paste("{title:'", xlabel,"', textStyle:{fontSize:'14'}}", sep=""), 
+                                                legend=paste("{textStyle:{fontSize:'12'}}", sep=""), 
+                                                chartArea="{left:80,top:50,width:\"75%\",height:\"80%\"}", 
+                                                bar=paste("{groupWidth:", gbar_width,"}", sep="")))
+          }
+      }else{
+          # Need to order the table before passing it to gviz
+          # Dont include the IP ddress in the sum!
+          if (ncol(de) > 2) { 
+            de$total <- rowSums(de[,-1], na.rm=TRUE);
+            de <-arrange(de, desc(total));
+            de$total <- NULL
+          } else {
+            de <-arrange(de, desc(de[[2]]));
+          }
+          # p <- gvisTable(de)
+          p <- gvisBarChart(de, xvar='x', yvar=y_var,
+                            options=list(isStacked=TRUE, title=title, height=600, width=940,
+                                         vAxis=paste("{title:'",xlabel,"',textStyle:{fontSize:'10'}}", sep=""),
+                                         hAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'14'}}", sep=""),
+                                         legend=paste("{textStyle:{fontSize:'12'}}", sep=""),
+                                         chartArea="{left:110,top:50,width:\"75%\",height:\"80%\"}"))
+      }
+      cat(p$html$chart,file=f)
+    }else{
+      if (scalex == "discrete") {
+        df$x <- factor(df$x, levels=rev(as.character(df$x)))
+        dfmelt <- melt(df)
+        de <- cast(dfmelt, x ~ key, value='value')
+        if (ncol(de) > 2) {
+          de$total <- rowSums(de, na.rm=TRUE)
+          dorder2 <- de[with(de, order(total)), ]
+          x_order <- dorder2$x
+          df$x <- factor(df$x, levels=x_order)
+        }
+      }
+
+      nKeys <- length(unique(df$key))
+      png(f, width = W, height = H)
+
+      p <- ggplot(data=df, aes(x=x, y=y, fill=key, order=key)) + 
+                  geom_bar(stat="identity") + 
+                  labs(title=title, x=xlabel, y=ylabel) + 
+                  scale_y_continuous(expand=c(0,0), labels = comma) +
+                  theme_bw() +
+                  theme(panel.grid.major = element_line(colour = GRIDGREY), 
+                        panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
+                  guides(fill = guide_legend(nrow = 20, byrow = TRUE))
+
+      if (nKeys <= NCBPALETTE) {
+        p <- p + scale_fill_manual(values=CBPALETTE)
+      }
+
+      if (scalex == "continuous") {
+        p <- p +
+             scale_x_continuous(expand=c(0.01,0), labels = comma)
+      }
+
+      if (vertical == 0) {
+        p <- p +
+             coord_flip() +
+             theme(panel.grid.major.x = element_line(colour = GRIDGREY), 
+                   panel.grid.minor.x = element_line(colour = GRIDGREY, linetype = "dotted"), 
+                   panel.grid.major.y = element_blank(), 
+                   panel.grid.minor.y = element_blank(),
+                   axis.text.y=element_text(size=14))
+      } else {
+          p <- p +
+               geom_bar(width=1, stat="identity")
+      }
+
+      if (pltnm == 'rcode_vs_replylen') {
+          p <- p + 
+               scale_x_continuous(expand=c(0.01,0), labels = comma, limits=c(0,1000))
+      }
+
+      if (pltnm == 'rcode_vs_replylen_big') {
+          xmin <- 1000
+          p <- p + 
+               scale_x_continuous(expand=c(0.01,0), labels = comma, limits=c(xmin,max(df$x) + 100))
+      }
+
+      print(p)
+      dev.off()
+    }
+}
+
+stackedAreaPlot <- function(df, f, title, xlabel, ylabel, gvis) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In stackedAreaPlot')
+  }
+  
+    if(gvis == 1){
+      title <- sub("\n", " ", title)
+      de <- cast(df, x ~ key, value='y', fun.aggregate=sum)
+      y_var <- tail(colnames(de),-1)
+      p <- gvisAreaChart(de, xvar='x', yvar=y_var, 
+                         options=list(isStacked=TRUE, title=title, height=600, width=940, 
+                                      vAxis=paste("{title:'",ylabel,"'}", sep=""), 
+                                      hAxis=paste("{title:'",xlabel,"'}", sep=""), 
+                                      legend=paste("{textStyle:{fontSize:'12'}}"), 
+                                      chartArea="{left:80,top:50,width:\"75%\",height:\"80%\"}"))
+      cat(p$html$chart,file=f)
+    }else{
+        df$x <- as.POSIXct(df$x)
+      nKeys <- length(unique(df$key))
+      png(f, width = W, height = H)
+
+      p <- ggplot(data=df, aes(x=x, y=y, fill=key, order=key)) + 
+                  geom_area(stat="identity") + 
+                  labs(title=title, x=xlabel, y=ylabel) +
+                  scale_x_datetime(expand=c(0.01,0)) +
+                  scale_y_continuous(expand=c(0,0), labels = comma) + 
+                  theme_bw() +
+                  theme(panel.grid.major = element_line(colour = GRIDGREY), 
+                        panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
+                  guides(fill = guide_legend(nrow = 20, byrow = TRUE))
+
+      if (nKeys <= NCBPALETTE) {
+        p <- p + scale_fill_manual(values=CBPALETTE)
+      }
+
+      print(p)
+      dev.off()
+    }
+}
+
+facetedBarPlot <- function(df, f, title, xlabel, ylabel, gvis, bar_width) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In facetedBarPlot')
+  }
+  
+    if(gvis == 1){
+        stackedBarPlot(df, f, title, xlabel, ylabel, gvis, pltnm = 'N/A', vertical = 1, gbar_width = bar_width)
+    }else{
+        nKeys <- length(unique(df$key))
+        png(f, width = W, height = H)
+
+        p <- ggplot(data=df, aes(x=x, y=y)) + 
+                    geom_bar(stat="identity", width=bar_width, fill=DARKERRED) +
+                    facet_grid(key ~ ., scales="free") + 
+                    labs(title=title, x=xlabel, y=ylabel) + 
+                    theme_bw() +
+                    theme(panel.grid.major = element_line(colour = GRIDGREY), 
+                          panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
+                    guides(fill = guide_legend(nrow = 20, byrow = TRUE))
+
+        if (nKeys <= NCBPALETTE) {
+          p <- p + scale_fill_manual(values=CBPALETTE)
+        }
+
+        print(p)
+        dev.off()
+    }
+}
+
+facetedLinePlot <- function(df, f, title, xlabel, ylabel, gvis) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In facetedLinePlot')
+  }
+
+    if(gvis == 1){
+      # fix up the legends to work with svg
+      rkey_svg <- sub("dns-(.*-.*-).*-(.*)", "\\1\\2", df$key)
+      rkey_svg <- sub("responses", "resp", rkey_svg)
+      df["key"] <- rkey_svg
+        linePlot(df, f, title, xlabel, ylabel, gvis)
+    }else{
+        nKeys <- length(unique(df$key))
+        png(f, width = W, height = H)
+        df$x <- as.POSIXct(df$x)
+        rkey1 <- sub("dns-(.*)-.*-.*-(.*)", "\\1-\\2", df$key)
+        df["rkey"] <- rkey1
+        rkey2 <- sub("dns-.*-(.*)-.*-.*", "\\1", df$key)
+        df["key"] <- rkey2
+
+        p <- ggplot(data=df, aes(x=x, y=y, group=key, colour=key)) +
+                    geom_jitter(position=position_jitter(width=2)) +
+                    geom_line() +
+                    labs(title=title, x=xlabel, y=ylabel) +
+                    facet_grid(rkey ~ ., scales="free") +
+                    scale_x_datetime(expand=c(0.01,0)) +
+                    scale_y_continuous(expand=c(0.01,0), labels=comma) +
+                    theme_bw() +
+                    theme(panel.grid.major = element_line(colour = GRIDGREY), 
+                          panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
+                    guides(col = guide_legend(nrow = 20, byrow = TRUE, override.aes=list(size=3)))
+
+        nKeys = length(unique(df$key))
+        if (nKeys <= NMINICBPALETTE) {
+            p <- p + scale_colour_manual(values=MINICBPALETTE)
+        }
+        else if (nKeys <= NCBPALETTE) {
+          p <- p + scale_colour_manual(values=CBPALETTE)
+        }
+
+        print(p)
+        dev.off()
+    }
+}
+
+facetedLinePlot2 <- function(df, f, title, xlabel, ylabel, gvis) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In facetedLinePlot2')
+  }
+
+    if(gvis == 1){
+        linePlot(df, f, title, xlabel, ylabel, gvis)
+    }else{
+        nKeys <- length(unique(df$key))
+        png(f, width = W, height = H)
+        df$x <- as.POSIXct(df$x)
+        df$facet1 <- sub(".*:tcp", "tcp", df$key)
+        df$facet2 <- sub(".*:udp", "udp", df$facet1)
+		write.table(df, file="/tmp/df.out")
+
+        p <- ggplot(data=df, aes(x=x, y=y, group=key, colour=key)) +
+                    geom_jitter(position=position_jitter(width=2)) +
+                    geom_line() +
+                    labs(title=title, x=xlabel, y=ylabel) +
+                    facet_grid(facet2 ~ ., scales="free") +
+                    scale_x_datetime(expand=c(0.01,0)) +
+                    scale_y_continuous(expand=c(0.01,0), labels=comma) +
+                    theme_bw() +
+                    theme(panel.grid.major = element_line(colour = GRIDGREY), 
+                          panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
+                    guides(col = guide_legend(nrow = 20, byrow = TRUE, override.aes=list(size=3)))
+
+        nKeys = length(unique(df$key))
+        if (nKeys <= NMINICBPALETTE) {
+            p <- p + scale_colour_manual(values=MINICBPALETTE)
+        }
+        else if (nKeys <= NCBPALETTE) {
+          p <- p + scale_colour_manual(values=CBPALETTE)
+        }
+
+        print(p)
+        dev.off()
+    }
+}
+
+facetedDiffLinePlot <- function(df, f, title, xlabel, ylabel, gvis) {
+
+    if (hh_debug) {
+      system('logger -p user.notice Hedgehog: In facetedDiffLinePlot')
+    }
+
+    nKeys <- length(unique(df$key))
+    df$x <- as.POSIXct(df$x)
+    rkey1 <- sub("dns-(.*)-.*-.*-(.*)", "\\1-\\2", df$key)
+    png(f, width=W, height=H)
+    df["rkey"] = rkey1
+
+    dfx <- arrange(df, x, rkey, desc(key))
+    dfx$y <- ifelse(test = (grepl("responses", dfx$key)), yes = -dfx$y, no = dfx$y)
+    df1 <- aggregate(dfx$y, by=list(x2=dfx$x, rkey=dfx$rkey), FUN=sum)
+    df1 <- plyr::rename(df1,c("x"="y", "x2"="x", "rkey"="key"))
+    if (gvis == 1) {
+        linePlot(df1, f, title, xlabel, ylabel, gvis)
+    } else {
+
+        p <- ggplot(data=df1, aes(x=x, y=y, group=key)) +
+                    geom_line(colour=DARKERRED) +
+                    labs(title=title, x=xlabel, y="Difference Between Number of Queries and Responses/min") +
+                    facet_grid(key ~ ., scales="free") +
+                    scale_x_datetime(expand=c(0.01,0)) +
+                    scale_y_continuous(expand=c(0.01,0), labels=comma) +
+                    theme_bw() +
+                    theme(panel.margin=grid::unit(3,"mm"), 
+                          panel.grid.major=element_line(colour=GRIDGREY),
+                          panel.grid.minor=element_line(colour=GRIDGREY, linetype="dotted")) +
+                    guides(col=guide_legend(nrow=20, byrow=TRUE, override.aes=list(size=3)))
+        if (nKeys <= NCBPALETTE) {
+            p <- p + scale_colour_manual(values=CBPALETTE)
+        }
+        print(p)
+        dev.off()
+    }
+}
+
+scatterPlot <- function(df, f, title, xlabel, ylabel, gvis) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In scatterPlot')
+  }
+
+    if(gvis == 1){
+      title <- sub("\n", " ", title)
+      dt <- df
+      #de <- as.data.frame(cast(df, serial + x + y ~ key, value='key'))
+      df$key <- NULL
+      df$x <- NULL
+      df$serial <- as.integer(df$serial)
+      df <- df[,c("serial", "y")]
+      df$y.html.tooltip <- paste(dt$key,": ", dt$y, " (", dt$x, ")",  sep="")
+      serial_min <- min(df$serial, na.rm = TRUE) - 1
+      serial_max <- max(df$serial, na.rm = TRUE) + 1
+      p <- gvisScatterChart(df, 
+                        options=list(legend="none", title=title, height=600, width=940, tooltip="{isHtml:'true'}, {textStyle: {fontSize: '18'}}",
+                                     vAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'10'}}", sep=""), 
+                                     hAxis=paste("{format:'', title:'",xlabel,"',textStyle:{fontSize:'14'}, viewWindow:{min:'",serial_min,"', max:'",serial_max,"'}}", sep=""), 
+                                     #hAxis=paste("{format:'', title:'",xlabel,"',textStyle:{fontSize:'14'}, ticks:[2014061900, 2014061901, 2014061902, 2014061903]}", sep=""),
+                                     chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
+      cat(p$html$chart,file=f)
+    }
+    else {
+      png(f, width = W, height = H)
+        df$key <- NULL
+        df$x <- NULL
+      p <- ggplot(data=df, aes(x=serial, y=y)) +
+                  geom_point(size = 4, stat="identity") +
+                  labs(title=title, x=xlabel, y=ylabel) +
+                  theme_bw() + scale_y_continuous(expand=c(0.05,0), labels = comma) +
+                  theme(panel.grid.major.y = element_line(colour = GRIDGREY), 
+                        panel.grid.minor.y = element_line(colour = GRIDGREY, linetype = "dotted"), 
+                        panel.grid.major.x = element_blank(), panel.grid.minor.x = element_blank())
+
+      print(p)
+      dev.off()
+    }
+}
+
+
+geomap <- function(df, f, title, xlabel, ylabel, gvis) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In geomap')
+    msg <- paste('logger -p user.notice file =', f)
+    system(msg)
+  }
+  if(gvis == 1){
+    title <- sub("\n", " ", title)
+    p <- gvisMap(df, locationvar='location', tipvar='sum', 
+                 options=list(colorAxis="{colors:['#FF0000', '#00FF00']}", displayMode='regions', height=600, width=940,
+                              chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
+    cat(p$html$chart,file=f)
+  }
+}
+
+geochart <- function(df, f, title, xlabel, ylabel, gvis) {
+
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In geomap')
+    msg <- paste('logger -p user.notice file =', f)
+    system(msg)
+  }
+  if(gvis == 1){
+    title <- sub("\n", " ", title)
+    p <- gvisGeoChart(df, locationvar='location', sizevar='count', colorvar='color', hovervar='type',
+                      options=list(displayMode='markers', height=600, width=940, colorAxis="{colors:['red','green']}", legend="'none'", tooltip="{showColorCode:false}",
+                                   chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
+    cat(p$html$chart,file=f)
+  }
+}
+
+getStmntParameters <- function(dsccon, dbdrv, dd_pltid, prepStmtNm, srvrid, start, stop) {
+
+  # Retrieving the datasets ids for the plot
+  if (!(prepStmnt("getdatasetids", dsccon))) {
+    return
+  }
+
+  sql <- paste("EXECUTE ", "getdatasetids", "('", dd_pltid, "');", sep="")
+
+  dataframe <- dbGetDataFrame(dbdrv, dsccon, dbconstr, sql)
+
+  if (is.null(dataframe) || ncol(dataframe) != 1) {
+    return
+  }
+
+  # Preparing the statement parameters with 1 or 2 datasets for the plot
+  ds_ids_list <- gsub("[{}]","",dataframe[1])
+  list <- strsplit(ds_ids_list,",")
+  num_ds_ids <- length(list[[1]])
+   
+  if ( num_ds_ids == 1 ) {
+    sql <- paste("EXECUTE ",prepStmtNm, "(", srvrid, ", '", list[[1]][1], "', timestamptz '", start, "', timestamptz '", stop, "');", sep="")
+  }
+  else if ( num_ds_ids == 2 ) {
+    sql <- paste("EXECUTE ",prepStmtNm, "(", srvrid, ", '", list[[1]][1], "', '", list[[1]][2], "', timestamptz '", start, "', timestamptz '", stop, "');", sep="")
+  }
+  else {
+    return
+  }    
+  return(sql)
+}
+
+generateYaml <- function(dsccon) {
+
+  if (hh_debug) {
+      system('logger -p user.notice Hedgehog: In generateYaml')
+    }
+
+  # Only allow yaml calls from local host in this implementation
+  # as this is only called from the cron script for now
+    if (SERVER$hostname != "localhost") 
+    return(FALSE)
+    
+  # Determine the plot being requested
+    prepStmnt("getpltdetails", dsccon)
+    pltdetails <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getpltdetails(", GET$pltid, ");", sep=""))
+    
+    if (is.null(pltdetails)) {
+        return(FALSE)
+    }   
+  pltnm     <- pltdetails$name
+
+    # Nasty hard coding because the traffic size plot is split into two
+  if (pltnm == 'traffic_sizes_small') {
+    pltnm = 'traffic_sizes'
+  } 
+  if (pltnm == 'traffic_sizes_big') {
+    return(TRUE)
+  } 
+  
+    prepStmnt("getsrvr_display_name_from_id", dsccon)
+    server_display_name <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getsrvr_display_name_from_id(", GET$svrid, ");", sep=""))
+
+  # Load the config file with the rssac spec in it 
+    rssac_config = yaml.load_file(paste(hh_config$directories$web_conf, "/rssac.yaml", sep=""))
+
+  # Define the base parameters for the yaml
+  metric <- sub("_", "-", pltnm)  
+  metric_data <- rssac_config$metric[[metric]]
+  # If the node looks like a root server, then set the service from the yaml
+  # otherwise, just use the server name
+  server_name_lc <- tolower(server_display_name)
+  server_name_root <- substr(server_name_lc,2,6)
+  if (server_name_root == "-root") {
+    server_service <- paste(substr(server_name_lc,1,1), metric_data$service, sep="")
+  } else {
+    server_service <- server_name_lc
+  }
+    date <- format(as.POSIXct(sub("T", " ", GET$start)), '%Y-%m-%d')
+
+  # Set up the output directory and file name
+    main_dir <- paste(hh_config$directories$rssac, "/", format(as.POSIXct(date), '%Y'), "/", format(as.POSIXct(date), '%m'), "/",  metric, "/", sep="")
+    dir.create(file.path(main_dir), showWarnings = FALSE, recursive = TRUE)
+    yaml_file <- paste(main_dir, server_name_lc, "-", format(as.POSIXct(date), '%Y%m%d'), "-", metric, ".yaml", sep="")
+
+    # Generate the query catering for the nodes specified
+    prepStmntNm <- pltnm
+    if (GET$ndarr == '-1') {
+        prepStmntNm <- paste(prepStmntNm, "_all_nodes", sep="")
+    }
+
+  sql <- getStmntParameters(dsccon, dbdrv, GET$pltid, prepStmntNm, GET$svrid, GET$start, GET$stop)
+  if(is.null(sql)){
+    return(FALSE)
+  }
+    
+  if (GET$ndarr != '-1'){
+        sql <- sub(");", paste(", '", GET$ndarr, "');", sep=""), sql)
+    }
+  
+  # Get the data frame
+    if (!prepStmnt(prepStmntNm, dsccon)) 
+    return(FALSE)
+    
+    df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, sql)
+
+  # TODO: Handle empty data frames better
+  # This simply means there isn't any data for this server on this day
+  if(is.null(df) || nrow(df) == 0){
+    return(TRUE)
+  }
+
+  # Now process the data from the data frame into yaml format
+    yaml_out <- list('version'='rssac002v3', service=server_service, 'start-period'=paste(GET$start, ":00Z", sep=""), metric=metric)
+
+    if (metric_data$statistics_type == "buckets") {
+        de <- as.data.frame(cast(df, x ~ key, value='y', fun.aggregate=sum))  
+        
+        for (s in metric_data$statistics) {
+            bucketlist <- list()
+            
+            for (i in seq(s$bucket_min, s$bucket_max, by=s$bucket_step)) {
+                bucket <- paste(i, "-", sep="")
+                
+                if (i != s$bucket_max) {
+                    bucket <- paste(bucket, i+s$bucket_step-1, sep="")
+                    value <- de[which(de$x == i + (s$bucket_step - 1)/2),s$name]
+                } 
+                else if (max(de$x) >= s$bucket_max) {
+                    value <- sum(subset(de, x >= s$bucket_max, select = s$name))
+                }
+
+                if (length(value)==0 || is.na(value) || is.null(value) || value==0) {
+                    bucketlist[[bucket]] <- as.integer(0)
+                } 
+                else {
+                    bucketlist[[bucket]] <- as.numeric(value)
+                }
+            }  
+            yaml_out[[s$name]] <- bucketlist
+        }
+    }
+  else if (metric_data$statistics_type == "kv_pair") {
+        de <- as.data.frame(cast(df, ~ key, value='y', fun.aggregate=sum))
+        for (s in metric_data$statistics) {
+            value <- as.integer(0)            
+            if (s %in% colnames(de)){
+                value <- de[[s]]
+            } 
+            yaml_out[[s]] <- value
+        }
+    } 
+  else if (metric_data$statistics_type == "kv_pair_with_alias") {
+        for (s in metric_data$statistics) {
+            value <- as.integer(0)
+            if (s$alias %in% df$x) {
+              value <- df[df$x == s$alias,2]
+      }
+            yaml_out[[s$name]] <- value
+        }
+  } 
+  else if (metric_data$statistics_type == "count") {
+    de <- as.data.frame(cast(df, ~ key, value='y', fun.aggregate=sum))
+    
+    for (s in metric_data$statistics) { 
+      countlist <- list()
+      for (i in colnames(de)) {
+        if (i != "value") {
+          countlist[[i]] <- de[[i]]
+        }
+      }
+      yaml_out[[s$name]] <- countlist
+    }
+  }
+  else if (metric_data$statistics_type == "average") {
+    df$x <- NULL
+    #de <- as.data.frame(cast(df, ~ serial, value='y', fun.aggregate=max))
+    de <- as.data.frame(cast(df, ~ serial, value='y', function(x) quantile(x,c(0.95))))
+    
+    for (s in metric_data$statistics) { 
+      countlist <- list()
+      for (i in colnames(de)) {
+        if (i != "value") {
+          countlist[[i]] <- as.integer(de[[i]])
+        }
+      }
+      yaml_out[[s$name]] <- countlist
+    }
+  }
+  else if (metric_data$statistics_type == "value") {
+    df$d.starttime <- NULL
+    for (s in metric_data$statistics) { 
+      countlist <- list()
+      for(i in 1:nrow(df))  {
+          countlist[df$x[[i]]] <- df$y[[i]]
+      }
+      yaml_out[[s$name]] <- countlist
+    }
+  }
+  
+  # Write the result to the yaml file
+  # A large precision is needed to deal with large (up to 64 bit) numbers
+    write(paste("---\n", gsub("\\.0\\n","\\\n", as.yaml(yaml_out,precision = 20)),sep=""), yaml_file)
+    return(TRUE)
+}
+
+# initialise plot options
+initPlotOptions <- function() {
+  
+  if (hh_debug) {
+    system('logger -p user.notice Hedgehog: In initPlotOptions')
+  }
+
+  # first, create the groups that link plots to the prepared statements
+  f1                      <<- c("edns_version", "do_bit", "client_port_range", "client_subnet2_trace")
+  f1lookupcodes           <<- c("qtype", "rcode", "dnssec_qtype")
+  f1lookupcodesnoquery    <<- c("opcode")
+  f1noclr                 <<- c("rd_bit")
+  f1count                 <<- c("client_subnet2_count")
+  f1nonormal              <<- c("idn_qname")
+  format1                 <<- c(f1, f1lookupcodes, f1lookupcodesnoquery, f1noclr, f1count, f1nonormal)
+    
+  f2mergekeys             <<- c("direction_vs_ipproto")
+  f2mergekeys_lookup      <<- c("certain_qnames_vs_qtype")
+  f2mergekeys_lookup_key1 <<- c("chaos_types_and_names")
+  f2other                 <<- c("server_addr_vs_trans")
+  f2sumkey2values         <<- c("pcap_stats", "transport_vs_qtype", "dns_ip_version")
+  format2                 <<- c(f2mergekeys, f2mergekeys_lookup, f2mergekeys_lookup_key1, f2sumkey2values)
+    
+  format3                 <<- c("client_subnet_accum", "ipv6_rsn_abusers_accum")
+  format3_bgpprefix       <<- c("bgpprefix")
+  format3_asn             <<- c("asn")
+  formattraffic           <<- c("traffic_volume", "traffic_volume_difference")
+
+  formatother             <<- c("qtype_vs_tld", "qtype_vs_legacygtld", "qtype_vs_cctld", "qtype_vs_newgtld", "qtype_vs_othertld", "client_addr_vs_rcode_accum", "qtype_vs_qnamelen", 
+                                "rcode_vs_replylen", "rcode_vs_replylen_big", "client_subnet2_accum", "dns_ip_version_vs_qtype", "by_node", "by_instance", "by_city", "by_country",
+                                "load_time", "zone_size", "server_addr", "client_subnet_vs_tld", "by_region")
+
+  rssac                   <<- c("traffic_volume", "traffic_sizes_small","traffic_sizes_big", "rcode_volume", "unique_sources", "traffic_volume_difference")
+  
+  geo                     <<- c("geomap", "geochart")
+  
+  formatother             <<- c(formatother, rssac, geo)
+  formatother             <<- c(formatother, "client_addr_vs_rcode_accum_asn", "client_addr_vs_rcode_accum_bgpprefix", "client_subnet2_accum_asn", "client_subnet2_accum_bgpprefix", "dnskey_vs_asn")
+
+  unknown_graphs          <<- c("client_subnet_count", "idn_vs_tld", "ipv6_rsn_abusers_count")
+
+  # now create other useful groups    
+  passplotname            <<- c(f1lookupcodes, f1lookupcodesnoquery)
+  avgoverwindow           <<- c(format3, format3_bgpprefix, format3_asn, 'qtype_vs_tld', 'qtype_vs_legacygtld', 'qtype_vs_cctld', 'qtype_vs_newgtld', 'qtype_vs_othertld',
+                                'client_addr_vs_rcode_accum', 'client_subnet2_accum', 'dns_ip_version_vs_qtype', 'client_addr_vs_rcode_accum_asn', 'client_addr_vs_rcode_accum_bgpprefix',
+                                "client_subnet2_accum_asn", "client_subnet2_accum_bgpprefix", "client_subnet_vs_tld", "dnskey_vs_asn")
+  lineplots               <<- c(format1, format2, "by_node", "by_instance", "by_city", "by_country", "rcode_volume", "server_addr", "by_region")
+  facetedbarplots         <<- c("traffic_sizes_small","traffic_sizes_big")
+  facetedlineplots        <<- c("traffic_volume")
+  facetedlineplots2       <<- c("server_addr_vs_trans")
+  faceteddifflineplots    <<- c("traffic_volume_difference")
+  log_option              <<- c(f1, f1lookupcodes, f1lookupcodesnoquery, f1noclr, f1nonormal, format2, "by_node", "by_instance", "by_city", "by_country", "rcode_volume", "server_addr", "by_region")
+}
+
+# create plot file if not cached
+generatePlotFile <- function(plttitle, pltnm, ddpltid, plot_file, simple_start, simple_stop, svrid, pltid, gvis, ndarr, dsccon) {
+  
+  if (hh_debug) {
+    msg <- paste('logger -p user.notice In generatePlotFile. Plot name =', pltnm, sep=" ")
+    system(msg)
+  }
+  if (pltnm == 'geomap' && gvis == 0) {
+    plot_file <- "plots/no_graph.png"
+    return(plot_file)
+  }
+  if (pltnm == 'geochart' && gvis == 0) {
+    plot_file <- "plots/no_graph.png"
+    return(plot_file)
+  }    
+    
+  mytitle <- paste(plttitle, "\nfrom ", simple_start, " UTC to ", simple_stop," UTC ", sep="")
+  time_window <- difftime(simple_stop, simple_start, units="secs")
+      
+  if (time_window > 86340 && pltnm %in% lineplots) {
+    mytitle <- paste(mytitle, " (smoothed)", sep="")
+  }
+
+  # Choose the prepared statement based on the group
+  prepStmntNm <- ""
+  if      (pltnm %in% f1)                         {prepStmntNm <- "f1"}
+  else if (pltnm %in% f1lookupcodes )             {prepStmntNm <- "f1lookupcodes"}
+  else if (pltnm %in% f1lookupcodesnoquery)       {prepStmntNm <- "f1lookupcodesnoquery"}
+  else if (pltnm %in% f1noclr)                    {prepStmntNm <- "f1noclr"}
+  else if (pltnm %in% f1count)                    {prepStmntNm <- "f1count"}
+  else if (pltnm %in% f1nonormal)                 {prepStmntNm <- "f1nonormal"}
+  else if (pltnm %in% f2mergekeys)                {prepStmntNm <- "f2mergekeys"}
+  else if (pltnm %in% f2mergekeys_lookup)         {prepStmntNm <- "f2mergekeys_lookup"}
+  else if (pltnm %in% f2mergekeys_lookup_key1)    {prepStmntNm <- "f2mergekeys_lookup_key1"}
+  else if (pltnm %in% f2sumkey2values)            {prepStmntNm <- "f2sumkey2values"}
+  else if (pltnm %in% f2other)                    {prepStmntNm <- "f2mergekeys"}
+  else if (pltnm %in% format3 )                   {prepStmntNm <- "format3"}
+  else if (pltnm %in% format3_bgpprefix )         {prepStmntNm <- "format3_bgpprefix"}
+  else if (pltnm %in% format3_asn )               {prepStmntNm <- "format3_asn"}  
+  else if (pltnm %in% formattraffic )             {prepStmntNm <- "traffic_volume"}
+  else if (pltnm %in% formatother)                {prepStmntNm <-  pltnm}
+
+  # Do other fix ups to the SQL before running the query
+  if (ndarr == '-1') {
+    prepStmntNm <- paste(prepStmntNm, "_all_nodes", sep="")
+  }
+  
+  if (!(prepStmnt(prepStmntNm, dsccon))) {
+    return
+  }
+  # Setting statement parameters with 1 or 2 datasets for the plot
+  sql <- getStmntParameters(dsccon, dbdrv, ddpltid, prepStmntNm, svrid, simple_start, simple_stop)
+  if (is.null(sql)) {
+    return
+  } 
+  if (pltnm %in% passplotname){
+    sql <- sub(");", paste(", '", pltnm, "');", sep=""), sql)
+  }
+  if (ndarr != '-1'){
+    sql <- sub(");", paste(", '", ndarr, "');", sep=""), sql)
+  }
+  if(pltnm %in% avgoverwindow){
+    sql <- sub("\\(", paste("\\(" , time_window, ".0, ", sep=""), sql)
+  }
+
+  # Get the data  **RUN THE QUERY **
+  df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, sql)
+
+    # Set the axis labels
+  xlab <- "Date"
+  if       (pltnm %in% format3)                   {xlab <- "Subnet (IPv4/8 or IPv6/32)"}
+  else if  (pltnm %in% format3_bgpprefix)         {xlab <- "Subnet (As seen in BGP)"}
+  else if  (pltnm %in% format3_asn)               {xlab <- "Subnet ASN"}
+  else if  (pltnm == 'traffic_sizes_small' ||
+            pltnm == 'traffic_sizes_big')         {xlab <- "Message Size (bytes)"}
+  else if (pltnm == 'zone_size')              {xlab <- "Serial Number"}
+  else if (pltnm == 'load_time')              {xlab <- "Serial Number"}
+    
+  ylab <- "Queries/sec"
+  if      (pltnm %in% rssac)                  {ylab <- "Queries/min"}
+  else if (pltnm %in% f1count)                {ylab <- "# Client Subnets"}
+  else if (pltnm %in% format3 ||
+           pltnm %in% format3_bgpprefix ||
+           pltnm %in% format3_asn)                {ylab <- "Average Query Rate (q/sec)"}
+  else if (pltnm == 'traffic_sizes_small' ||
+             pltnm == 'traffic_sizes_big')      {ylab <- "Number of Queries in Each 16 Byte Group"}
+  else if (pltnm == 'zone_size')              {ylab <- "Zone Size (Bytes)"}
+  else if (pltnm == 'load_time')              {ylab <- "Zone Propagation Time (Seconds)"}
+    
+  # Now decide how to plot it
+  if      (is.null(df))                               {plot_file <- "plots/no_connection.png"}
+  else if (nrow(df) == 0)                             {plot_file <- "plots/no_results.png"}
+  else if ((nrow(df) == 1) && (pltnm %in% lineplots)) {plot_file <- "plots/no_results.png"}
+  else if (pltnm %in% f1count)                        {stackedAreaPlot     (df, plot_file, mytitle, xlab, ylab, gvis)}
+  else if (pltnm %in% lineplots)                      {linePlot            (df, plot_file, mytitle, xlab, ylab, gvis)}
+  else if (pltnm %in% facetedlineplots)               {facetedLinePlot     (df, plot_file, mytitle, xlab, ylab, gvis)}
+  else if (pltnm %in% facetedlineplots2)              {facetedLinePlot2    (df, plot_file, mytitle, xlab, ylab, gvis)}
+  else if (pltnm %in% faceteddifflineplots)           {facetedDiffLinePlot (df, plot_file, mytitle, xlab, ylab, gvis)}
+  else if (pltnm %in% facetedbarplots)                {facetedBarPlot      (df, plot_file, mytitle, xlab, ylab, gvis, 14)} # width hardcoded to 14
+  else if (pltnm %in% format3 ||
+           pltnm %in% format3_bgpprefix ||
+           pltnm %in% format3_asn)                        {barPlot             (df, plot_file, mytitle, xlab, ylab, gvis)}
+  else if (pltnm == 'qtype_vs_tld' || 
+           pltnm == 'qtype_vs_legacygtld' || 
+           pltnm == 'qtype_vs_cctld' || 
+           pltnm == 'qtype_vs_newgtld' ||
+           pltnm == 'qtype_vs_othertld')             {stackedBarPlot(df, plot_file, mytitle, "TLD",                        "Average Query Rate (q/sec)", gvis, pltnm)}
+  else if (pltnm == 'client_addr_vs_rcode_accum' || 
+           pltnm == 'client_subnet2_accum' ||
+           pltnm == 'client_subnet_vs_tld')           {stackedBarPlot(df, plot_file, mytitle, "Subnet (IPv4/8 or IPv6/32)", "Average Query Rate (q/sec)", gvis, pltnm)}
+  else if (pltnm == 'client_addr_vs_rcode_accum_asn') {stackedBarPlot(df, plot_file, mytitle, "Subnet by ASN", "Average Query Rate (q/sec)", gvis, pltnm)}
+  else if (pltnm == 'client_addr_vs_rcode_accum_bgpprefix') {stackedBarPlot(df, plot_file, mytitle, "Subnet as seen in BGP", "Average Query Rate (q/sec)", gvis, pltnm)}
+  else if (pltnm == 'client_subnet2_accum_asn')       {stackedBarPlot(df, plot_file, mytitle, "Subnet by ASN", "Average Query Rate (q/sec)", gvis, pltnm)}
+  else if (pltnm == 'client_subnet2_accum_bgpprefix')       {stackedBarPlot(df, plot_file, mytitle, "Subnet as seen in BGP", "Average Query Rate (q/sec)", gvis, pltnm)}
+
+  else if (pltnm == 'dnskey_vs_asn')                  {stackedBarPlot(df, plot_file, mytitle, "Count of DNSKEY Queries by ASN", "Average Query Rate (q/sec)", gvis, pltnm)}
+    
+  else if (pltnm == 'qtype_vs_qnamelen')              {stackedBarPlot(df, plot_file, mytitle, "QNAME Length (bytes)",       "Count",                      gvis, pltnm, scalex="continuous", vertical=1)}
+
+  else if (pltnm == 'dns_ip_version_vs_qtype')        {stackedBarPlot(df, plot_file, mytitle, "IP Version",                 "Average Query Rate (q/sec)", gvis, pltnm, vertical=1)}
+  else if (pltnm == 'unique_sources')                 {barPlot       (df, plot_file, mytitle, "IP Version/Aggregation",     "Number of Unique Sources",   gvis, small=0)}
+  else if (pltnm == 'rcode_vs_replylen' || 
+           pltnm == 'rcode_vs_replylen_big')          {if (gvis==1) {stackedAreaPlot(df, plot_file, mytitle, "Response Size Length (bytes)", "Count", gvis)}
+                                                       else         {stackedBarPlot (df, plot_file, mytitle, "Response Size Length (bytes)", "Count", gvis, pltnm, scalex="continuous", vertical=1)}}
+  else if (pltnm == 'geomap')                         {geomap  (df, plot_file, mytitle, "aaaa", "bbbbb", 1)}
+  else if (pltnm == 'geochart')                       {geochart(df, plot_file, mytitle, "aaaa", "bbbbb", 1)}
+  else if (pltnm == 'load_time')                      {scatterPlot(df, plot_file, mytitle, xlab, ylab, gvis)}
+  else if (pltnm == 'zone_size')                      {barPlot(df, plot_file, mytitle, xlab, ylab, gvis, vertical=1)}
+
+  return(plot_file)
+}
diff --git a/www/hedgehog/www/R/prepareStatement.R b/www/hedgehog/www/R/prepareStatement.R
new file mode 100755
index 0000000..b56465f
--- /dev/null
+++ b/www/hedgehog/www/R/prepareStatement.R
@@ -0,0 +1,629 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+#TODO(refactor): Tidy this up so common strings are re-used and we don't need separate node and all_node versions of the same query
+
+prepStmnt <- function(statementNm, dsccon){
+
+    if(class(dsccon) != "try-error"){
+      select_nodes_sql="AND node_id = ANY (string_to_array($6, ',')::integer[])"
+      all_nodes_prep="(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS"
+      select_nodes_prep="(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS"
+      skipped_sql="NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')"
+      
+        rs <- NULL
+        switch(statementNm,
+               getdbversion                         = { rs <- try(dbSendQuery(dsccon, "PREPARE getdbversion AS                      SELECT version FROM dsc.version;"))},
+               getpltddcategories                   = { rs <- try(dbSendQuery(dsccon, "PREPARE getpltddcategories AS                SELECT DISTINCT ddcategory, cast(substring(ddcategory from 1 for position('.' in ddcategory)-1 ) as int) as in_order FROM dsc.plot WHERE ddcategory != '' order by in_order;"))},
+               getpltid_ddname                      = { rs <- try(dbSendQuery(dsccon, "PREPARE getpltid_ddname (TEXT) AS            SELECT id, ddname FROM dsc.plot WHERE ddcategory=$1 ORDER BY ddname ;"))},
+               getpltdetails                        = { rs <- try(dbSendQuery(dsccon, "PREPARE getpltdetails (INTEGER) AS           SELECT name, title, description, plot_id FROM dsc.plot WHERE id=$1;"))},
+               getsrvrid_display_name               = { rs <- try(dbSendQuery(dsccon, "PREPARE getsrvrid_display_name AS            SELECT id, display_name FROM dsc.server ORDER BY display_name;"))},
+               getsrvr_display_name_from_id         = { rs <- try(dbSendQuery(dsccon, "PREPARE getsrvr_display_name_from_id AS      SELECT display_name FROM dsc.server where id=$1;"))},
+               getgroups                            = { rs <- try(dbSendQuery(dsccon, "PREPARE getgroups (INTEGER) AS               SELECT DISTINCT CASE WHEN (region='' OR region IS NULL) THEN 'Other' ELSE region END FROM dsc.node WHERE server_id=$1 ORDER BY region;"))},
+
+               getnodes                             = { rs <- try(dbSendQuery(dsccon, "PREPARE getnodes         (INTEGER) AS        SELECT id, name, CASE WHEN (country='' OR country IS NULL) THEN 'No Country' ELSE country END as country, CASE WHEN (city='' OR city IS NULL) THEN 'No City' ELSE city END as city, CASE WHEN (subgroup='' OR subgroup IS NULL) THEN 'No Instance' ELSE subgroup END as subgroup FROM dsc.node WHERE server_id=$1 ORDER BY name;"))},
+               getnodesbyregion                     = { rs <- try(dbSendQuery(dsccon, "PREPARE getnodesbyregion (INTEGER, TEXT) AS  SELECT id, name, CASE WHEN (country='' OR country IS NULL) THEN 'No Country' ELSE country END as country, CASE WHEN (city='' OR city IS NULL) THEN 'No City' ELSE city END as city, CASE WHEN (subgroup='' OR subgroup IS NULL) THEN 'No Instance' ELSE subgroup END as subgroup FROM dsc.node WHERE server_id=$1 AND region=$2 ORDER BY name;"))},
+
+               getdatasetids                        = { rs <- try(dbSendQuery(dsccon, "PREPARE getdatasetids    (INTEGER) AS        SELECT dataset_id FROM dsc.plot WHERE id=$1;"))},
+               getdefaultpltid                      = { rs <- try(dbSendQuery(dsccon, "PREPARE getdefaultpltid  (TEXT) AS           SELECT id FROM dsc.plot WHERE name=$1;"))},
+
+               by_node                              = { rs <- try(dbSendQuery(dsccon, "PREPARE by_node                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, n.name AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_node_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE by_node_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, n.name AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+
+               by_region                            = { rs <- try(dbSendQuery(dsccon, "PREPARE by_region                            (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.region   = '' OR n.region   IS NULL) THEN 'Other'       ELSE n.region   END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_region_all_nodes                  = { rs <- try(dbSendQuery(dsccon, "PREPARE by_region_all_nodes                  (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.region   = '' OR n.region   IS NULL) THEN 'Other'       ELSE n.region   END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_instance                          = { rs <- try(dbSendQuery(dsccon, "PREPARE by_instance                          (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.subgroup = '' OR n.subgroup IS NULL) THEN 'No Instance' ELSE n.subgroup END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_instance_all_nodes                = { rs <- try(dbSendQuery(dsccon, "PREPARE by_instance_all_nodes                (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.subgroup = '' OR n.subgroup IS NULL) THEN 'No Instance' ELSE n.subgroup END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_city                              = { rs <- try(dbSendQuery(dsccon, "PREPARE by_city                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.city     = '' OR n.city     IS NULL) THEN 'No City'     ELSE n.city     END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_city_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE by_city_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.city     = '' OR n.city     IS NULL) THEN 'No City'     ELSE n.city     END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_country                           = { rs <- try(dbSendQuery(dsccon, "PREPARE by_country                           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.country  = '' OR n.country  IS NULL) THEN 'No Country'  ELSE n.country  END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               by_country_all_nodes                 = { rs <- try(dbSendQuery(dsccon, "PREPARE by_country_all_nodes                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.country  = '' OR n.country  IS NULL) THEN 'No Country'  ELSE n.country  END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+
+               f1                                   = { rs <- try(dbSendQuery(dsccon, "PREPARE f1                                   (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1_all_nodes                         = { rs <- try(dbSendQuery(dsccon, "PREPARE f1_all_nodes                         (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1lookupcodes                        = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodes                        (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT, TEXT) AS     select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IN ('FormErr', 'NotImp', 'YXDomain', 'YXRRSet', 'NotAuth', 'NotZone', 'BADKEY', 'BADTIME', 'BADMODE', 'BADNAME', 'BADALG', 'BADTRUNC', 'BADVERS') THEN 'Other' WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1lookupcodes_all_nodes              = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodes_all_nodes              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IN ('FormErr', 'NotImp', 'YXDomain', 'YXRRSet', 'NotAuth', 'NotZone', 'BADKEY', 'BADTIME', 'BADMODE', 'BADNAME', 'BADALG', 'BADTRUNC', 'BADVERS') THEN 'Other' WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1lookupcodesnoquery                 = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodesnoquery                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) AND d.key1 !='Query' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1lookupcodesnoquery_all_nodes       = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodesnoquery_all_nodes       (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.key1 !='Query' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1noclr                              = { rs <- try(dbSendQuery(dsccon, "PREPARE f1noclr                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND key1 !='clr' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1noclr_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE f1noclr_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1 !='clr' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1count                              = { rs <- try(dbSendQuery(dsccon, "PREPARE f1count                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value) AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1count_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE f1count_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value) AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1nonormal                           = { rs <- try(dbSendQuery(dsccon, "PREPARE f1nonormal                           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND key1 !='normal' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f1nonormal_all_nodes                 = { rs <- try(dbSendQuery(dsccon, "PREPARE f1nonormal_all_nodes                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1 !='normal' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2mergekeys                          = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys                          (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2mergekeys_all_nodes                = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_all_nodes                (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2mergekeys_lookup                   = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup                   (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry::text='qtype' WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2mergekeys_lookup_all_nodes         = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup_all_nodes         (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || CASE WHEN il.name IS NULL then 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry::text='qtype' WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2mergekeys_lookup_key1              = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup_key1              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, il.name || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, iana_lookup il WHERE il.registry::text='qtype' AND key1::text = il.value::text AND d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2mergekeys_lookup_key1_all_nodes    = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup_key1_all_nodes    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, il.name || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, iana_lookup il WHERE il.registry::text='qtype' AND key1::text = il.value::text AND d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2sumkey2values                      = { rs <- try(dbSendQuery(dsccon, "PREPARE f2sumkey2values                      (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               f2sumkey2values_all_nodes            = { rs <- try(dbSendQuery(dsccon, "PREPARE f2sumkey2values_all_nodes            (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
+               format3                              = { rs <- try(dbSendQuery(dsccon, "PREPARE format3                        (REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           SELECT dsc.iptruncate(key2::ipaddress) AS x, sum(value)/$1 AS y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND key2 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x ORDER BY y DESC LIMIT 40;"))},
+               format3_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE format3_all_nodes              (REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 SELECT dsc.iptruncate(key2::ipaddress) AS x, sum(value)/$1 AS y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x ORDER BY y DESC LIMIT 40;"))},
+
+               client_addr_vs_rcode_accum           = { rs <- try(dbSendQuery(dsccon, "PREPARE client_addr_vs_rcode_accum               (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS  SELECT dsc.iptruncate(key1::ipaddress) AS x, il.name AS key, sum(d.value)/$1 As y FROM dsc.data d, iana_lookup il WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'rcode' AND d.key2::text = il.value::text AND dsc.iptruncate(key1::ipaddress) IN (SELECT dsc.iptruncate(key1::ipaddress) AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
+               client_addr_vs_rcode_accum_all_nodes = { rs <- try(dbSendQuery(dsccon, "PREPARE client_addr_vs_rcode_accum_all_nodes     (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ) AS        SELECT dsc.iptruncate(key1::ipaddress) AS x, il.name AS key, sum(d.value)/$1 As y FROM dsc.data d, iana_lookup il WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'rcode' AND d.key2::text = il.value::text AND dsc.iptruncate(key1::ipaddress) IN (SELECT dsc.iptruncate(key1::ipaddress) AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
+               qtype_vs_qnamelen                    = { rs <- try(dbSendQuery(dsccon, "PREPARE qtype_vs_qnamelen                     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT il.name AS key, cast(d.key2 AS int) AS x, sum(d.value) AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'qtype' AND d.key1::text = il.value::text AND cast(d.key2 AS int) < 100 GROUP BY key, x;"))},
+               qtype_vs_qnamelen_all_nodes          = { rs <- try(dbSendQuery(dsccon, "PREPARE qtype_vs_qnamelen_all_nodes           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT il.name AS key, cast(d.key2 AS int) AS x, sum(d.value) AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'qtype' AND d.key1::text = il.value::text AND cast(d.key2 AS int) < 100 GROUP BY key, x;"))},
+               rcode_vs_replylen                    = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen                     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer < 1000;"))},
+               rcode_vs_replylen_all_nodes          = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen_all_nodes           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer < 1000;"))},
+               rcode_vs_replylen_big                = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen_big                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer >= 1000;"))},
+               rcode_vs_replylen_big_all_nodes      = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen_big_all_nodes       (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer >= 1000;"))},
+               client_subnet2_accum                 = { rs <- try(dbSendQuery(dsccon, "PREPARE client_subnet2_accum                     (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS  SELECT dsc.iptruncate(key1::ipaddress) AS x, key2 AS key, sum(value)/$1 As y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND substring(key1 FROM '^([0-9]*.)|.*:.*') IN (SELECT substring(key1 FROM '^([0-9]*.)|.*:.*')::text AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
+               client_subnet2_accum_all_nodes       = { rs <- try(dbSendQuery(dsccon, "PREPARE client_subnet2_accum_all_nodes           (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ) AS        SELECT dsc.iptruncate(key1::ipaddress) AS x, key2 AS key, sum(value)/$1 As y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND substring(key1 FROM '^([0-9]*.)|.*:.*') IN (SELECT substring(key1 FROM '^([0-9]*.)|.*:.*')::text AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
+               dns_ip_version_vs_qtype              = { rs <- try(dbSendQuery(dsccon, "PREPARE dns_ip_version_vs_qtype                  (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS  SELECT d.key1 AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 As y FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype' WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x, key ORDER BY y DESC;"))},
+               dns_ip_version_vs_qtype_all_nodes    = { rs <- try(dbSendQuery(dsccon, "PREPARE dns_ip_version_vs_qtype_all_nodes        (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ) AS        SELECT d.key1 AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 As y FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype' WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x, key ORDER BY y DESC;"))},
+
+               traffic_volume                       = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_volume                   (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT starttime AS x, 'dns-' || key1 || '-queries-received-' || lower(key2) AS key, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY x, key UNION SELECT starttime AS x, 'dns-' || key1 || '-responses-sent-' || lower(key2) AS key, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY x, key;"))},
+               traffic_volume_all_nodes             = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_volume_all_nodes         (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT starttime AS x, 'dns-' || key1 || '-queries-received-' || lower(key2) AS key, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 GROUP BY x, key UNION SELECT starttime AS x, 'dns-' || key1 || '-responses-sent-' || lower(key2) AS key, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 GROUP BY x, key;"))},
+               # Following two statements are used to generate yaml data, but not for a visible plot
+               traffic_sizes                        = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes                    (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5  AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x ORDER BY x;"))},
+               traffic_sizes_all_nodes              = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_all_nodes          (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data   WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 GROUP BY key, x ORDER BY x;"))},
+               traffic_sizes_small                  = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_small              (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x ORDER BY x;"))},
+               traffic_sizes_small_all_nodes        = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_small_all_nodes    (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data   WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 GROUP BY key, x ORDER BY x;"))},
+               traffic_sizes_big                    = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_big                (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x ORDER BY x;"))},
+               traffic_sizes_big_all_nodes          = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_big_all_nodes      (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data   WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 GROUP BY key, x ORDER BY x;"))},
+               rcode_volume                         = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_volume                     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT d.starttime AS x, d.key1 AS key, sum(d.value) AS y FROM dsc.data d WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY x, key;"))},
+               rcode_volume_all_nodes               = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_volume_all_nodes           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT d.starttime AS x, d.key1 AS key, sum(d.value) AS y FROM dsc.data d WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY x, key;"))},
+               unique_sources_raw                   = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources_raw               (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT key1 as x, count(key2) AS y FROM dsc.data as d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY x UNION SELECT 'IPv6/64' as x, count(*) as y from (SELECT substring(key2 FROM '(^([0-9a-f]{1,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}))') as subnet FROM dsc.data as d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1='IPv6' AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY subnet) AS sq ORDER BY y DESC;"))},
+               unique_sources_raw_all_nodes         = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources_raw_all_nodes     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT key1 as x, count(key2) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY x UNION SELECT 'IPv6/64' as x, count(*) as y from (SELECT substring(key2 FROM '(^([0-9a-f]{1,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}))') as subnet FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1='IPv6' GROUP BY subnet) AS sq ORDER BY y DESC;"))},
+               unique_sources                       = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources                   (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT key1 as x, sum(value) AS y FROM dsc.data as d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY x order by y DESC;"))},
+               unique_sources_all_nodes             = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources_all_nodes         (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT key1 as x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY x order by y desc;"))},
+               load_time                            = { rs <- try(dbSendQuery(dsccon, "PREPARE load_time                        (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT d.starttime AS x, n.name AS key, d.value AS y, d.key2 as serial FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) order by serial;"))},
+               load_time_all_nodes                  = { rs <- try(dbSendQuery(dsccon, "PREPARE load_time_all_nodes              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT d.starttime AS x, n.name AS key, d.value AS y, d.key2 as serial FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 order by serial;"))},
+               zone_size                            = { rs <- try(dbSendQuery(dsccon, "PREPARE zone_size                        (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT d.starttime, avg(d.value) AS y, d.key2 AS x FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) group by d.starttime, d.key2 order by x;"))},
+               zone_size_all_nodes                  = { rs <- try(dbSendQuery(dsccon, "PREPARE zone_size_all_nodes              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT d.starttime, avg(d.value) AS y, d.key2 AS x FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 group by d.starttime, d.key2 order by x;"))},
+
+               geomap = {
+                 sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS
+                 select g.latitude || ':' || g.longitude as location, sum(d.value) 
+                 FROM dsc.data d, geoip g where cast(d.key2 as ipaddress) <<= g.ip_range 
+                 and server_id=$1 AND plot_id=$2 AND d.starttime >=$3 
+                 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) 
+                 and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+                 group by location;", sep=" ")
+                sql=gsub("\n"," ",sql_joined)
+                rs <- try(dbSendQuery(dsccon, sql))},
+               
+               geomap_all_nodes = {
+                 sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS
+                 select g.latitude || ':' || g.longitude as location, sum(d.value) 
+                 FROM dsc.data d, geoip g 
+                 where cast(d.key2 as ipaddress) <<= g.ip_range 
+                 and server_id=$1 AND plot_id=$2 AND d.starttime >=$3 
+                 AND starttime<=$4 and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+                 group by location;", sep=" ")
+                sql=gsub("\n"," ",sql_joined)
+                rs <- try(dbSendQuery(dsccon, sql))},
+               
+               geochart = {
+                 sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS
+                 select l.name as location, sum(d.value) as count, 1 as color, 'Queries' as type 
+                 FROM dsc.data d, geoip g, locations l  
+                 where cast(d.key2 as ipaddress) <<= g.ip_range 
+                 and l.geoname_id=g.geoname and server_id=$1 AND plot_id=$2 
+                 AND d.starttime >=$3 AND starttime<=$4 
+                 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) 
+                 and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+                 group by location 
+                 union 
+                 select n.city as location, count(*), 2 as color, n.city || ' nodes' as type 
+                 from node n 
+                 where n.id = ANY (string_to_array($5, ',')::integer[])
+                 and server_id=$1 and n.city!='' and n.city IS NOT NULL
+                 group by n.city;", sep=" ")
+                sql=gsub("\n"," ",sql_joined)
+                rs <- try(dbSendQuery(dsccon, sql))},
+               
+               geochart_all_nodes = {
+                 sql_joined <- paste("PREPARE", statementNm,  "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS
+                 select l.name as location, sum(d.value) as count, 1 as color, 'Queries' as type 
+                 FROM dsc.data d, geoip g, locations l  
+                 where cast(d.key2 as ipaddress) <<= g.ip_range 
+                   and l.geoname_id=g.geoname and server_id=$1 AND plot_id=$2 
+                   AND d.starttime >=$3 AND starttime<=$4 
+                   and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+                 group by location 
+                 union 
+                 select n.city as location, count(*), 2 as color, n.city || ' nodes' as type 
+                 from node n
+                 where server_id=$1 and n.city!='' and n.city IS NOT NULL
+                 group by n.city;", sep=" ")
+                sql=gsub("\n"," ",sql_joined)
+                rs <- try(dbSendQuery(dsccon, sql))},
+
+               format3_bgpprefix  = {
+                 sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
+                 "SELECT xx / ip2bgpprefix(xx) AS x, sum(y) as y FROM
+                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y 
+                  FROM dsc.data 
+                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                  select_nodes_sql,
+                  "AND key2", skipped_sql,
+                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
+                group by x order by y desc;", sep=" ")
+                sql=gsub("\n"," ",sql_joined)
+                rs <- try(dbSendQuery(dsccon, sql))},
+               
+               format3_bgpprefix_all_nodes= {
+                 sql_joined <- paste("PREPARE", statementNm, all_nodes_prep, 
+                 "SELECT xx / ip2bgpprefix(xx) AS x, sum(y) as y FROM 
+                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y
+                   FROM dsc.data
+                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                   AND key2", skipped_sql,
+                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
+                group by x order by y desc;", sep=" ")
+                sql=gsub("\n"," ",sql_joined)
+                rs <- try(dbSendQuery(dsccon, sql))},
+
+               format3_asn = {
+                 sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
+                 "SELECT ip2asn(xx) AS x, sum(y) as y FROM
+                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y 
+                   FROM dsc.data 
+                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                   select_nodes_sql,
+                   "AND key2", skipped_sql,
+                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
+                group by x order by y desc;", sep=" ")
+                sql=gsub("\n"," ",sql_joined)
+                rs <- try(dbSendQuery(dsccon, sql))},
+               
+               format3_asn_all_nodes = {
+                 sql_joined <- paste("PREPARE", statementNm, all_nodes_prep, 
+                 "SELECT ip2asn(xx) AS x, sum(y) as y FROM 
+                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y 
+                   FROM dsc.data 
+                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                   AND key2", skipped_sql,
+                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
+                group by x order by y desc;", sep=" ")
+                 sql=gsub("\n"," ",sql_joined)
+                 rs <- try(dbSendQuery(dsccon, sql))},
+               
+               client_addr_vs_rcode_accum_asn = {
+               sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
+                 "SELECT dsc.ip2asn(xx) AS x, key, sum(y) as y FROM 
+                  (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
+                   FROM dsc.data d, iana_lookup il 
+                   WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                   AND d.key1", skipped_sql,
+                   "AND il.registry = 'rcode' AND d.key2::text = il.value::text",
+                   select_nodes_sql,
+                   "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+              sql=gsub("\n"," ",sql_joined)
+              rs <- try(dbSendQuery(dsccon, sql))},
+              
+              client_addr_vs_rcode_accum_asn_all_nodes = {
+              sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
+                "SELECT dsc.ip2asn(xx) AS x, key, sum(y) as y FROM 
+                 (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
+                  FROM dsc.data d, iana_lookup il 
+                  WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                  AND d.key1 ", skipped_sql,
+                  "AND il.registry = 'rcode' AND d.key2::text = il.value::text
+                  GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+             sql=gsub("\n"," ",sql_joined)
+             rs <- try(dbSendQuery(dsccon, sql))},
+ 
+             client_addr_vs_rcode_accum_bgpprefix = {
+             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
+               "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM 
+                (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
+                 FROM dsc.data d, iana_lookup il 
+                 WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                 AND d.key1", skipped_sql,
+                 "AND il.registry = 'rcode' AND d.key2::text = il.value::text",
+                 select_nodes_sql,
+                 "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+            sql=gsub("\n"," ",sql_joined)
+            rs <- try(dbSendQuery(dsccon, sql))},
+            
+            client_addr_vs_rcode_accum_bgpprefix_all_nodes = {
+            sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
+              "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM 
+               (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
+                FROM dsc.data d, iana_lookup il 
+                WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                AND d.key1", skipped_sql,
+                  "AND il.registry = 'rcode' AND d.key2::text = il.value::text
+                GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           client_subnet2_accum_asn = {
+             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
+               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
+                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
+                  FROM dsc.data 
+                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                  select_nodes_sql,
+                  "AND key1", skipped_sql,
+                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+           
+           client_subnet2_accum_asn_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
+               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
+                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
+                  FROM dsc.data 
+                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                  "AND key1", skipped_sql,
+                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           client_subnet2_accum_bgpprefix = {
+             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
+               "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM
+                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
+                  FROM dsc.data 
+                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                  select_nodes_sql,
+                  "AND key1", skipped_sql,
+                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+           
+           client_subnet2_accum_bgpprefix_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
+               "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM
+                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
+                  FROM dsc.data 
+                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                  "AND key1", skipped_sql,
+                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},               
+
+           server_addr = {
+             sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
+               "select to_timestamp((extract(epoch from sq.sx)::int /
+                 (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*
+                 (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x,
+                sq.skey as key, avg(sq.sy) as y from
+                  (SELECT d.starttime AS sx, d.key1 AS skey, sum(d.value)/60.0 AS sy
+                   FROM dsc.data d
+                   WHERE d.server_id=$1
+                    AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4
+                    AND d.node_id = ANY (string_to_array($5, ',')::integer[])
+                    AND (d.key1::ipaddress IN (SELECT address FROM service_addr WHERE server_id=$1) OR (SELECT count(*) FROM service_addr WHERE server_id=$1) = 0)
+                   GROUP BY sx, skey) as sq
+                GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+
+           server_addr_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
+             "select to_timestamp((extract(epoch from sq.sx)::int /
+               (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*
+               (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x,
+              sq.skey as key, avg(sq.sy) as y from
+                (SELECT d.starttime AS sx, d.key1 AS skey,
+                sum(d.value)/60.0 AS sy 
+                FROM dsc.data d
+                WHERE d.server_id=$1 AND d.plot_id=$2
+                  AND d.starttime>=$3 AND d.starttime<=$4
+                  AND (d.key1::ipaddress IN (SELECT address FROM service_addr WHERE server_id=$1) OR (SELECT count(*) FROM service_addr WHERE server_id=$1) = 0)
+                GROUP BY sx, skey) as sq
+              GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           # client_subnet_vs_tld = {
+           #   sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
+           #   "SELECT dsc.iptruncate(key1::ipaddress)::text AS x, key2 AS key, sum(value)/$1 As y
+           #   FROM dsc.data
+           #   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
+           #     AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+           #     AND node_id = ANY (string_to_array($6, ',')::integer[])
+           #     AND dsc.iptruncate(key1::ipaddress) IN
+           #     (SELECT dsc.iptruncate(key1::ipaddress) AS k1
+           #       FROM dsc.data
+           #       WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4
+           #       AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[])
+           #       AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+           #       GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40)
+           #  GROUP BY x, key;", sep=" ")
+           # sql=gsub("\n"," ",sql_joined)
+           # rs <- try(dbSendQuery(dsccon, sql))},
+           #
+           # client_subnet_vs_tld_all_nodes = {
+           #   sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
+           #   "SELECT dsc.iptruncate(key1::ipaddress)::text AS x, key2 AS key, sum(value)/$1 As y
+           #   FROM dsc.data
+           #   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
+           #     AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+           #     AND dsc.iptruncate(key1::ipaddress) IN
+           #     (SELECT dsc.iptruncate(key1::ipaddress) AS k1
+           #       FROM dsc.data
+           #       WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4
+           #       AND starttime<=$5
+           #       AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+           #     GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40)
+           #   GROUP BY x, key;", sep=" ")
+           # sql=gsub("\n"," ",sql_joined)
+           # rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_tld = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
+             "SELECT lower(d.key1) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
+             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5
+               AND d.node_id = ANY (string_to_array($6, ',')::integer[])
+               AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+               AND d.key1 IN
+                 (SELECT key1
+                   FROM dsc.data 
+                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                   AND node_id = ANY (string_to_array($6, ',')::integer[])
+                   AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+                   GROUP BY key1 ORDER BY sum(value)/$1 DESC LIMIT 40)
+             GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_tld_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS", 
+             "SELECT lower(d.key1) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
+             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND d.key1 IN
+               (SELECT key1
+                 FROM dsc.data 
+                 WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
+                 AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+                 GROUP BY key1 ORDER BY sum(value)/$1 DESC LIMIT 40)
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_cctld = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
+             "SELECT alabel2ulabel(t.alabel) as x,
+             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t
+             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3
+             AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+             AND lower(d.key1) IN
+               (SELECT lower(dd.key1)
+                 FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
+                   LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
+                 WHERE nn.type_name='ccTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
+                 AND dd.starttime>=$4 AND dd.starttime<=$5
+                 AND dd.node_id = ANY (string_to_array($6, ',')::integer[])
+                 AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+                 GROUP BY lower(dd.key1)
+                 ORDER BY sum(value)/$1 DESC LIMIT 40)
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_cctld_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
+             "SELECT alabel2ulabel(t.alabel) as x,
+             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
+             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
+             AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
+             AND lower(d.key1) IN 
+               (SELECT lower(dd.key1)
+               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
+                 LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
+               WHERE nn.type_name='ccTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
+               AND dd.starttime>=$4 AND dd.starttime<=$5 
+               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+               GROUP BY lower(dd.key1) 
+               ORDER BY sum(value)/$1 DESC LIMIT 40) 
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_newgtld = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
+             "SELECT alabel2ulabel(t.alabel) as x,
+             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t
+             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
+             AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+             AND lower(d.key1) IN 
+               (SELECT lower(dd.key1) 
+                 FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
+                 LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
+                 WHERE nn.type_name='New-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
+                 AND dd.starttime>=$4 AND dd.starttime<=$5 
+                 AND dd.node_id = ANY (string_to_array($6, ',')::integer[]) 
+                 AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+                 GROUP BY lower(dd.key1) 
+                 ORDER BY sum(value)/$1 DESC LIMIT 40) 
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_newgtld_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
+             "SELECT alabel2ulabel(t.alabel) as x,
+             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
+             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
+             AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+             AND lower(d.key1) IN 
+               (SELECT lower(dd.key1) 
+               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
+               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
+               WHERE nn.type_name='New-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
+               AND dd.starttime>=$4 AND dd.starttime<=$5 
+               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+               GROUP BY lower(dd.key1) 
+               ORDER BY sum(value)/$1 DESC LIMIT 40) 
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_legacygtld = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
+             "SELECT alabel2ulabel(t.alabel) as x, 
+             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
+             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
+             AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+             AND lower(d.key1) IN 
+               (SELECT lower(dd.key1) 
+               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = tt.alabel 
+               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
+               WHERE nn.type_name='Legacy-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
+               AND dd.starttime>=$4 AND dd.starttime<=$5 
+               AND dd.node_id = ANY (string_to_array($6, ',')::integer[]) 
+               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+               GROUP BY lower(dd.key1) ORDER BY sum(value)/$1 DESC LIMIT 40) 
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_legacygtld_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
+             "SELECT alabel2ulabel(t.alabel) as x,
+             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
+             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
+             AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+             AND lower(d.key1) IN 
+               (SELECT lower(dd.key1) 
+               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
+               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
+               WHERE nn.type_name='Legacy-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
+               AND dd.starttime>=$4 AND dd.starttime<=$5 
+               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+               GROUP BY lower(dd.key1) 
+               ORDER BY sum(value)/$1 DESC LIMIT 40) 
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_othertld = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
+             "SELECT alabel2ulabel(lower(d.key1)) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
+             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+             AND lower(d.key1) IN 
+               (SELECT lower(dd.key1) 
+               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = tt.alabel 
+               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id WHERE nn.type_name is NULL 
+               AND dd.server_id=$2 AND dd.plot_id=$3 
+               AND dd.starttime>=$4 AND dd.starttime<=$5 
+               AND dd.node_id = ANY (string_to_array($6, ',')::integer[]) 
+               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+               GROUP BY lower(dd.key1) 
+               ORDER BY sum(value)/$1 DESC LIMIT 40) 
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           qtype_vs_othertld_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
+             "SELECT alabel2ulabel(lower(d.key1)) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
+             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
+             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 
+             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+             AND lower(d.key1) IN 
+               (SELECT lower(dd.key1) 
+               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
+               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id WHERE nn.type_name is NULL 
+               AND dd.server_id=$2 AND dd.plot_id=$3 
+               AND dd.starttime>=$4 AND dd.starttime<=$5 
+               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
+               GROUP BY lower(dd.key1) 
+               ORDER BY sum(value)/$1 DESC LIMIT 40) 
+            GROUP BY x, key;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+           dnskey_vs_asn = {
+             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
+               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
+                 (SELECT key2::ipaddress AS xx, key1 AS key, sum(value)/$1 AS y
+                  FROM dsc.data 
+                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                  select_nodes_sql,
+                  "AND key2", skipped_sql, "AND key1='48'",
+                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+           
+           dnskey_vs_asn_all_nodes = {
+             sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
+               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
+                 (SELECT key2::ipaddress AS xx, key1 AS key, sum(value)/$1 AS y
+                  FROM dsc.data 
+                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
+                  "AND key2", skipped_sql, "AND key1='48'",
+                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
+                group by x, key order by y desc;", sep=" ")
+           sql=gsub("\n"," ",sql_joined)
+           rs <- try(dbSendQuery(dsccon, sql))},
+
+
+        )
+
+        if(class(rs) == "try-error"){
+            system(paste('logger -p user.crit database failed to load prepared statement name: "', statementNm, '" FROM dbconn.R', sep=""))
+            return(FALSE)
+        }else{
+            return(TRUE)
+        }
+    }else{
+        return(FALSE)
+    }
+}
diff --git a/www/hedgehog/www/R/startup.R.in b/www/hedgehog/www/R/startup.R.in
new file mode 100755
index 0000000..d4695a4
--- /dev/null
+++ b/www/hedgehog/www/R/startup.R.in
@@ -0,0 +1,40 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+#system('logger -p user.notice Hedgehog: Call to Hedgehog startup.R')
+
+# Set the locale - For some reason this is set in R on the command line
+# but not in Rapache!
+Sys.setlocale("LC_ALL", 'en_US.UTF-8')
+
+suppressPackageStartupMessages(library(Cairo))
+suppressPackageStartupMessages(library(brew))
+suppressPackageStartupMessages(library(DBI))
+suppressPackageStartupMessages(library(RPostgreSQL))
+suppressPackageStartupMessages(library(ggplot2))
+suppressPackageStartupMessages(library(googleVis))
+suppressPackageStartupMessages(library(reshape))
+suppressPackageStartupMessages(library(RPostgreSQLHelper))
+suppressPackageStartupMessages(library(digest))
+suppressPackageStartupMessages(library(scales))
+suppressPackageStartupMessages(library(yaml))
+suppressPackageStartupMessages(library(plyr))
+suppressPackageStartupMessages(library(grid))
+suppressPackageStartupMessages(library(rjson))
+
+dbdrv  <- dbDriver("PostgreSQL")
+hh_config <- yaml.load_file("@HH_CONF_FILE@")
+gui_config <- yaml.load_file(paste(hh_config$directories$conf, "/hedgehog_gui.yaml", sep=""))
+
+
+# sink("/temp/R/R.out", append=TRUE)
+# print(as.list(.GlobalEnv))
+# sink()
diff --git a/www/hedgehog/www/R/utilities.R b/www/hedgehog/www/R/utilities.R
new file mode 100755
index 0000000..83ceaa4
--- /dev/null
+++ b/www/hedgehog/www/R/utilities.R
@@ -0,0 +1,67 @@
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+# 
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+handleTimeframe <- function() {
+	
+	if (hh_debug) {
+		system('logger -p user.notice Hedgehog: In handleTimeFrame')
+	}
+    
+	# Ensure data being pulled back is no more recent than presentation_delay_in_hours ago
+	lag_end <- format(Sys.time() - (gui_config$www$presentation_delay_in_hours*60*60), '%Y-%m-%dT%H:%M')
+	
+	if (GET$stop > lag_end) {
+		start_shift <- as.numeric(difftime(as.POSIXct(sub("T", " ", GET$stop)),as.POSIXct(sub("T", " ", GET$start)),units="hours"))
+		GET$stop    <<- lag_end
+		GET$start   <<- format(as.POSIXct(sub("T", " ", GET$stop)) - (start_shift*60*60), '%Y-%m-%dT%H:%M')
+	}
+}
+
+# Check that all calls to this function have the required parameters
+validateGetParams <- function() {
+	
+	if (hh_debug) {
+		system('logger -p user.notice Hedgehog: In validateGetParams')
+	}
+	
+	if (is.null(GET$type)) 
+		return(FALSE)
+
+    # Check that all calls to this function have the required parameters
+	if (GET$type == "generatePlot" || GET$type == "generateYaml") {
+		if (is.null(GET$start) || is.null(GET$stop)) {
+			system('logger -p user.notice Hedgehog: Invalid time')
+			return(FALSE)
+		}
+		if (is.null(GET$pltid)) 	 {
+			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid pltid')
+			return(FALSE)
+		}
+		if (is.null(GET$svrid))		 {
+			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid svrid')
+			return(FALSE)
+		}
+		if (is.null(GET$gvis))			 {
+			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid gvis')
+			return(FALSE)
+		}
+		if (is.null(GET$svrnm)) 				 {
+			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid svrnm')
+			return(FALSE)
+		}
+		if (is.null(GET$ndarr))					 {
+			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid ndarr')
+			return(FALSE)
+		}
+		handleTimeframe()
+	}   
+	return(TRUE)
+}
\ No newline at end of file
diff --git a/www/hedgehog/www/about.html.in b/www/hedgehog/www/about.html.in
new file mode 100644
index 0000000..f3d1397
--- /dev/null
+++ b/www/hedgehog/www/about.html.in
@@ -0,0 +1,123 @@
+<!--
+Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+
+This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0. If a copy of the MPL was not distributed with this
+file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+Developed by Sinodun IT (www.sinodun.com)
+-->
+
+<!DOCTYPE html>
+<!--[if lt IE 7 ]><html class="ie ie6" lang="en"> <![endif]-->
+<!--[if IE 7 ]><html class="ie ie7" lang="en"> <![endif]-->
+<!--[if IE 8 ]><html class="ie ie8" lang="en"> <![endif]-->
+<!--[if (gte IE 9)|!(IE)]><!--><html lang="en"> <!--<![endif]-->
+<head>
+
+	<!-- Basic Page Needs
+  ================================================== -->
+	<meta charset="utf-8">
+	<title>DNS-STATS: About Hedgehog</title>
+	<meta name="Hedgehog" content="">
+	<meta name="jad@sinodun.com" content="">
+
+	<!-- Mobile Specific Metas
+  ================================================== -->
+	<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
+
+	<!-- CSS
+  ================================================== -->
+	<link rel="stylesheet" href="stylesheets/base.css?version=@HH_PACKAGE_VERSION@">
+	<link rel="stylesheet" href="stylesheets/skeleton.css?version=@HH_PACKAGE_VERSION@">
+	<link rel="stylesheet" href="stylesheets/layout.css?version=@HH_PACKAGE_VERSION@">
+
+	<!--[if lt IE 9]>
+		<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
+	<![endif]-->
+
+	<!-- Favicons
+   ================================================== -->
+	<link rel="shortcut icon" href="images/favicon.ico">
+	<link rel="apple-touch-icon" href="images/apple-touch-icon.png">
+	<link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
+	<link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
+        
+</head>
+    <body>
+        <div class="container">
+            <div class="sixteen columns">
+                </br>
+                <h2 class="remove-bottom">About DNS-STATS:Hedgehog</h2>
+                <h5>Version @HH_PACKAGE_VERSION@: @HH_VERSION_NAME@ release</h5>
+                <hr style='margin-bottom:10px;' />
+                    <a href="hedgehog.html">DNS-STATS:Hedgehog</a> is a visualisation tool for DNS statistics that consumes data acquired with the 
+                    <a href="http://dns.measurement-factory.com/tools/dsc/">DSC</a> collector. <br/> Hedgehog was initially developed for 
+                    <a href="http://www.icann.org/">ICANN</a> by <a href="http://sinodun.com/">Sinodun IT</a> and is now released open 
+                    source under the Mozilla Public License, version 2.0. For more information on the development and support of DNS-STATS:Hedgehog see 
+                    the <a href="http://dns-stats.org/">dns-stats</a> website.<br>
+                    The DNS-STATS:Hedgehog software releases are named after <a href="https://en.wikipedia.org/wiki/Hedgehog#Genera_and_species">subspecies</a> of hedgehogs.
+                <hr />
+                <ul>
+                    <h5>Quick Help</h5></br>
+                    <li><strong>Plots</strong>
+                        <ul class="discs">
+                            <li>The paradigm in Hedgehog is that the user selects a set of criteria (plot, time window, 
+                                plot type and node selection) and then hits the 'Generate Plot!' button. (It is not the case
+                                that changing e.g. the plot or node selection immediately re-generates the current plot.)</li>
+                            <li>The static plots are .png files. Hedgehog offers the option to view these plots as normal
+                                line plots, stacked line plots or line plots with a logarithmic y-axis.</li>
+                            <li>The interactive plots are based on GoogleViz and provide zoom controls (fixed intervals 
+                                at the top left and a slider bar at the bottom) and display data values along the top as the
+                                user moves their mouse over the plot window.</li>
+                            <li>Plots are cached as they are viewed. The 'day' plots (midnight-midnight) for the previous
+                                day are cached overnight.</li>
+                            <li>For line plots of less than or equal to one day interval the data points are 1 minute 
+                                apart by default. Where the time interval is greater than 1 day the data is averaged to improve
+                                the appearance of the plot. These averaged plots contain a fixed number of data points which 
+                                results in the interval between the data points being 7 minutes for a 'week' plot and ~30 minutes
+                                for a 'month' plot.</li>
+                        </ul>
+                    </li>
+                    <li><strong>Time</strong>
+                        <ul class="discs">
+                            <li>The basic time controls generate graphs for fixed intervals. 
+                                <ul class="circles">
+                                    <li>The '&lt;' and '&gt;' buttons will shift the window forward or backwards by one interval unit.</li>
+                                    <li>The '+&lt;' and '&gt;+' buttons will increase the window width by one interval unit. </li>
+                                </ul>
+                            </li>
+                            <li>The advanced time controls allow the user to specify the time interval down to the minute.</li>
+                        </ul>
+                    </li>
+                    <li><strong>Node selection</strong>
+                        <ul class="discs">
+                            <li>The nodes are displayed in different tabs for each top level Group, including an 'All' group.</li> 
+                            <li>The nodes may be further grouped for display/selection purposes by 
+                                either Country/City/Instance or not grouped at all. This grouping is controlled via the radio selection next to
+                                the server drop down. This selection does not affect how nodes are grouped in plots; choose a different plot
+                                under the 'Node Statistics' category to change the grouping in plots.</li>
+                            <li>Within each Group tab, bulk node selections can be altered with the 'Actions' buttons.
+                                The icons next to the Group/Country/City/Instance names indicate if all (full circle), some (half circle) or none (empty circle) of the nodes
+                                in that choice are included in nodes for plot generation. Clicking on an individual Country/City/Instance/Node selects or de-selects that choice. </li>
+                        </ul>
+                    </li>
+                    <li><strong>Static link</strong>
+                         <ul class="discs">
+                             <li>The static link provided below the plot window is a link to a cached version of the plot file on the Hedgehog server. 
+                             This link may not be valid if the cache has been flushed since the plot was originally produced.</li>
+                         </ul>
+                     </li>
+                </ul>
+            </div>
+            <div class="sixteen columns" id="copyright" >
+                <hr />
+                &copy; <a href="http://www.icann.org/">Internet Corporation for Assigned Names and Numbers</a> <br/>Developed by <a href="http://sinodun.com/">Sinodun Internet Technologies</a>.
+            
+              <hr style='margin-top:5px;' />
+              This product includes GeoLite2 data created by MaxMind, available from
+              <a href="http://www.maxmind.com">http://www.maxmind.com</a>.
+            </div>
+        </div>
+    </body>
+</html>
diff --git a/www/hedgehog/www/brew/hedgehog.brew.in b/www/hedgehog/www/brew/hedgehog.brew.in
new file mode 100644
index 0000000..ffb0a81
--- /dev/null
+++ b/www/hedgehog/www/brew/hedgehog.brew.in
@@ -0,0 +1,471 @@
+<%
+# 
+# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+#
+# Developed by Sinodun IT (www.sinodun.com)
+#
+
+# Debug mode set to 1, set to 0 to disable
+hh_debug <<- 0
+
+# TODO(refactor): preparing and executing DB queries should be wrapped better and in a dbQuery.R file
+# TODO: use either tabs or spaces for indentation - we have both at the moment
+
+# Initialise the static/interactive default type
+initPltType <- function() {
+	
+	if (hh_debug) {
+	    system('logger -p user.notice Hedgehog: In initPltType')
+    }
+
+    %><%=gui_config$www$default_plot_type%><%
+}
+
+# Initialise the Plot drop down menu contents
+initPlotDDHtml <- function() {
+	
+	if (hh_debug) {
+	    system('logger -p user.notice Hedgehog: In initPlotDDHtml')
+    }
+
+	# TODO: These queries should only return plots that are configured for display
+	# Especially as not everyone will want the RSSAC plots showing
+    if (!(prepStmnt("getpltddcategories", dsccon)) || !(prepStmnt("getpltid_ddname", dsccon))){
+        %><option value="DB error">DB error preparing plot type query</option><%
+        return(FALSE)
+    }
+   
+    df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getpltddcategories;")
+ 
+    if (is.null(df)) {
+        %><option value="DB error">DB connection error</option><%
+        return(FALSE)
+    } 
+	else if (nrow(df) < 1) {
+    	%><option value="DB error">No plots found in DB</option><%
+        return(FALSE)
+    }
+	
+	for (i in 1:nrow(df)) {
+		df2 <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getpltid_ddname('", df[i,1], "');", sep=""))
+		if (is.null(df2)) {
+			%><option value="DB error">DB connection error</option><%
+			return(FALSE)
+		} 
+		else if (nrow(df2) < 1) {
+			%><option value="DB error">No data found in DB</option><%
+			return(FALSE)
+		}
+		optgroup_html <- sub(".*\\.","",df[i,1])
+		%><optgroup label="<%=optgroup_html%>"><%
+		for (i in 1:nrow(df2)) {
+                if (! is.null(gui_config$www$enable_node_selection)) {
+                        if ((gui_config$www$enable_node_selection == 0) && (df2[i,2] == "By node")) {
+                                next
+                        }
+                } else {
+                        system('logger -p user.warning Hedgehog: Missing enable_node_selection in config file. Using 1 enabled.')
+                }
+			%><option value="<%=df2[i,1]%>"><%=df2[i,2]%></option><%
+		}
+		%></optgroup><%
+	}
+    return(TRUE)
+}
+
+# Initialise the server drop down menu contents
+initServerDDHtml <- function() {
+	
+	if (hh_debug) {
+	    system('logger -p user.notice Hedgehog: In initServerDDHtml')
+    }
+
+    if (!(prepStmnt("getsrvrid_display_name", dsccon))) {
+        %><option value="DB error">DB error preparing server query</option><%
+        return(FALSE)
+    }
+    
+    df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getsrvrid_display_name;")
+    
+    if (is.null(df)) {
+        %><option value="DB error">DB connection error</option><%
+        return(FALSE)
+    } 
+	else if (nrow(df) < 1) {
+    	%><option value="DB error">No servers found in DB</option><%
+        return(FALSE)
+    }
+    
+    for (i in 1:nrow(df)) {
+		%><option value="<%=df[i,1]%>"><%=df[i,2]%></option><%
+	}
+    return(TRUE)
+}
+
+getDefaultServer <- function() {
+  
+  if (hh_debug) {
+      system('logger -p user.notice Hedgehog: In getDefaultServer')
+  }
+  
+    # Calling code checks for no server specified
+  %><%=gui_config$www$default_server%><%
+
+  return(TRUE)
+}
+
+# Generate a nested JSON structure the js code can use to create the node tabs
+initNodeData <- function() {
+    if (hh_debug) {
+        system('logger -p user.notice Hedgehog: In initNodeTabsHtml')
+    }
+
+    if (!(prepStmnt("getsrvrid_display_name", dsccon)) || !(prepStmnt("getgroups", dsccon)) || !(prepStmnt("getnodes", dsccon)) || !(prepStmnt("getnodesbyregion", dsccon))) {
+        %>DB error preparing node query<%
+        return(FALSE)
+    }
+
+    server=list()
+    servers_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getsrvrid_display_name;")
+    if (!is.null(servers_df) && nrow(servers_df) >= 1) {
+        for (s in 1:nrow(servers_df)) {
+            group_list=list()
+            srvr_id <- servers_df[s,1]
+            srvr_nm <- servers_df[s,2]
+            groups_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getgroups(", srvr_id, ");", sep=""))
+            
+            for (g in 0:nrow(groups_df)) {
+                nodes=list()
+                if (g == 0) {
+                    gp_nm = 'All'
+                    nodes_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getnodes(", srvr_id, ");", sep=""))
+                } else {
+                    gp_nm <- groups_df[g,1]
+                    # Other should really appear at the end of the list......
+                    if  (gp_nm == 'Other'){
+                        nodes_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getnodesbyregion(", srvr_id, ",'');", sep=""))
+                    } else {
+                        nodes_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getnodesbyregion(", srvr_id, ", '", gp_nm,"');", sep=""))
+                    }
+                }
+                for (n in 1:nrow(nodes_df)) {
+                    # Query fills in blank fields so we dont need to check here
+                    nd_id      = nodes_df[n,1]
+                    nd_nm      = nodes_df[n,2]
+                    nd_country = nodes_df[n,3]
+                    nd_city    = nodes_df[n,4]
+                    nd_sg      = nodes_df[n,5]
+                    node  = list(node_name=nd_nm, node_id=nd_id, node_country=nd_country, node_city=nd_city, node_sg=nd_sg)
+                    nodes = c(nodes, list(node))
+                }
+                group=list(group_name=gp_nm, node_list=nodes)
+                group_list=c(group_list, list(group))
+            }
+            server_list=list(server=srvr_nm, groups=group_list)
+            server=c(server, list(server_list))
+        }
+        node_json = rjson::toJSON(server)
+        output = cat(node_json)
+        %><%output%><%
+        return(TRUE)
+    }
+    %>No data returned<%
+    return(FALSE)
+}
+
+getDefaultPlotId <- function() {
+	
+	if (hh_debug) {
+	    system('logger -p user.notice Hedgehog: In getDefaultPlotId')
+	}
+	
+	# Query the db to get the default plot id
+	if (!(prepStmnt("getdefaultpltid", dsccon))) {
+		return(FALSE)
+	}
+	
+	defaultpltnm <- gui_config$www$default_plot_name
+	if (is.null(defaultpltnm)) {
+		system('logger -p user.warning Hedgehog: Missing Default Plot name in config file. Using By region.')
+		defaultpltnm <- "by_region"
+	}
+	defaultpltid <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getdefaultpltid('", defaultpltnm, "');", sep=""))	
+	
+	if (is.null(defaultpltid) || nrow(defaultpltid) == 0) {
+		return(FALSE)
+	}
+	%><%=as.integer(defaultpltid[1])%><%
+}
+
+getEnableNodeSelection <- function() {
+        if (hh_debug) {
+            system('logger -p user.notice Hedgehog: In getEnableNodeSelection')
+        }
+        ens <- gui_config$www$enable_node_selection
+        if (is.null(ens)) {
+                system('logger -p user.warning Hedgehog: Missing enable_node_selection in config file. Using 1 enabled.')
+                ens <- 1
+        }
+        %><%=ens%><%
+}
+
+getDefaultGrouping <- function() {
+	
+	if (hh_debug) {
+	    system('logger -p user.notice Hedgehog: In getDefaultGrouping')
+	}
+	
+	grouping = gui_config$www$default_node_grouping
+	if (is.null(grouping)) {
+		system('logger -p user.warning Hedgehog: Missing Node Grouping in config file. Using None.')
+		%>none<%
+		return(TRUE)
+	}
+
+	if      (grouping == "instance") {%>instance<%} 
+	else if (grouping == "city")     {%>city<%}
+	else if (grouping == "country")  {%>country<%} 
+	else if (grouping == "none")     {%>none<%}
+	else {
+		system('logger -p user.warning Hedgehog: Unsupported Node Grouping present in config file. Using Instance.')
+		%>instance<%
+	}
+	return(TRUE)
+}
+
+getSupportURL <- function() {
+	
+	if (hh_debug) {
+	    system('logger -p user.notice Hedgehog: In getSupportURL')
+	}
+
+    # Calling code checks for missing url
+	%><%=gui_config$www$support_url%><%
+
+	return(TRUE)
+}
+
+generatePlot <- function(){
+
+    #TODO(refactor): Move more of this code out to the generatePlot.R file
+
+	if(hh_debug){
+		system('logger -p user.notice Hedgehog: In generatePlot')
+	}
+
+	source(paste(hh_config$directories$web_source, "/R/generatePlot.R", sep=""))
+    
+	# Query the db to get the plot attributes
+	if (!(prepStmnt("getpltdetails", dsccon))) {
+		return(FALSE)
+	}
+
+	dd_pltid			<- strtoi(GET$pltid)
+	pltdetails			<- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getpltdetails(", dd_pltid, ");", sep=""))	
+	
+	if (is.null(pltdetails)) {
+		return(FALSE)
+	}
+	
+	pltnm				<- pltdetails$name
+	plttitle			<- pltdetails$title
+	pltdescription		<- pltdetails$description
+	pltid				<- pltdetails$plot_id
+		
+	svrid				<- GET$svrid
+	gvis				<- GET$gvis
+	ndarr				<- GET$ndarr
+	svrnm				<- GET$svrnm
+	start				<- GET$start
+	stop				<- GET$stop
+	
+	# Check the parameters
+	if (!(prepStmnt("getsrvrid_display_name", dsccon))) {
+		return(FALSE)
+	}
+	serverlist <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getsrvrid_display_name;")
+	if (is.null(serverlist)) {
+		return(FALSE)
+	}
+	check <- grep(svrnm,serverlist$display_name)
+	if (length(check) == 0) {
+		return(FALSE)
+	}
+	sid <- serverlist$id[check]
+	if (sid != svrid){
+		return(FALSE)
+	}
+		
+	# Initialise plot parameters
+	initPlotOptions()
+	
+	
+	# Initialise name of plot file and make sure the cache directory exists
+	plot_dir = paste(hh_config$directories$plotcache, "/", svrnm, "/", pltnm, sep="") 
+	dir.create(file.path(plot_dir), showWarnings = FALSE, recursive = TRUE)
+	if (pltnm %in% unknown_graphs ) {
+		plot_file = "plots/no_graph.png"
+	} 
+	else {
+		plot_file = paste(plot_dir, "/", start, "-", stop, "-all.png", sep="")
+		if (gvis == 1) {
+			plot_file <- sub(".png", ".html", plot_file)
+		}
+		if (ndarr != '-1') {
+			plot_file <- sub("-all.", paste("-", digest(ndarr, algo="sha256"), ".", sep=""), plot_file)
+		}
+	}
+
+	# convert the times to POSIX string
+	simple_start   <- as.POSIXct(start, format='%Y-%m-%dT%H:%M')
+	simple_stop    <- as.POSIXct(stop, format='%Y-%m-%dT%H:%M')
+
+	cache       <- FALSE # Create new plot_file
+	if (file.exists(plot_file) && gui_config$www$use_plot_caching == 1) {
+		lag_hours <- as.numeric(difftime(file.info(plot_file)$mtime,as.POSIXct(simple_start),units="hours"))
+		if (lag_hours > gui_config$www$caching_delay_in_hours) {
+			cache <- TRUE # Pull back file from disk
+		}
+	}
+
+	if (!cache) {	
+		plot_file <- generatePlotFile(plttitle, pltnm, dd_pltid, plot_file, simple_start, simple_stop, svrid, pltid, gvis, ndarr, dsccon)
+	}
+    
+	# OK - we have a plot file so now lets generate the html to hold that
+	now <- format(Sys.time(), '%Y-%m-%dT%H:%M')
+	
+	plot_file_raw = gsub('../','/',plot_file)
+	if (plot_file != "plots/no_graph.png" && plot_file != "plots/no_results.png" && plot_file != "plots/no_connection.png") {
+		plot_file <- sub(hh_config$directories$plotcache, "../plotcache", plot_file)
+        
+		if (gvis == '1') {
+			%><div id="plot">
+				<hr />
+				<iframe scrolling="no" width="100%" height="600px" frameborder="0" id="plotframe" src="<%=plot_file%>?nocache=<%=now%>"></iframe>
+			</div><%
+		}
+		else if (pltnm %in% log_option) {
+			log_file <- sub(".png", "-log.png", plot_file)
+			stack_file <- sub(".png", "-stack.png", plot_file)
+			%><div id="opts" class="" style="height=10px">
+				<hr style="margin-bottom:10px;"/>
+				<input type="radio" name="pltoptions" id="rbliny" onclick="toggle_plot()" title="Line plot with linear y-axis" checked>
+				<label for="rbliny" title="Line plot with linear y-axis">Linear scale</label>
+				<input type="radio" name="pltoptions" id="rblogy" onclick="toggle_plot()" title="Line plot with log y-axis">
+				<label for="rblogy" title="Line plot with log y-axis">Log scale</label>
+				<input type="radio" name="pltoptions" id="rbstack" onclick="toggle_plot()" title="Stacked line plot with linear y-axis">
+				<label for="rbstack"  title="Stacked line plot with linear y-axis">Stacked linear</label>
+			</div>
+			<div id="plot">
+				<img src="<%=plot_file%>?nocache=<%=now%>" id= "liny" name="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',plot_file)%>"/>
+				<img src="<%=log_file%>?nocache=<%=now%>" class="hidden" id="logy" name="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',log_file)%>" />
+				<img src="<%=stack_file%>?nocache=<%=now%>" class="hidden" id="stack" name="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',stack_file)%>"/>
+			</div><%
+		}
+		else {
+			%><div id="plot">
+				<hr />
+				<img id="staticplot" src="<%=plot_file%>?nocache=<%=now%>" />
+			</div><%
+		}
+		%><div id="plink" class="" style="height:27px; width:960px;">
+			<label style="font-size:80%" for="ploturi">Static Link</label>
+			<input type="text" id="ploturi" readonly="readonly" value="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',plot_file)%>" style="width: 870px; height:7px; font-size:80%">
+		</div><%
+	}
+	else {
+		%><div id="plot">
+			<hr />
+			<img src="<%=plot_file%>" />
+		</div><%
+	}
+}
+
+validateDbVersion <- function() {
+	
+	if (hh_debug) {
+		system('logger -p user.notice Hedgehog: In validateDbVersion')
+	}
+
+	if (!(prepStmnt("getdbversion", dsccon))) {
+		%>Error: preparing db query<%
+		return(FALSE)
+	}
+		
+	dbversion <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getdbversion;")
+    
+	if (is.null(dbversion)) {
+		%>Error: DB connection error<%
+		return(FALSE)
+	}
+	
+	if (dbversion != DB_SCHEMA_VERSION) {
+		%>Error: Database version mismatch. Contact an administrator.<%
+		return(FALSE)
+	}	
+
+	return(TRUE)
+}
+
+#########################################################
+###########  Entry point for brew calls #################
+#########################################################
+
+if (hh_debug) {
+    system('logger -p user.notice Hedgehog: Call to Hedgehog brew')
+}
+
+source(paste(hh_config$directories$web_source, "/R/utilities.R", sep = ""))
+if (!validateGetParams()) {
+	%>Error: invalid parameters in method call.<%
+	return()
+}
+
+if (GET$type == "initPltType") {
+	# this doesn't need to call the DB
+	initPltType()
+	return()
+}
+
+if (is.null(hh_config$database$read_pass)) { 
+    db_password <- ""
+} else {
+    db_password <- hh_config$database$read_pass
+}
+dsccon <- try(dbConnect(dbdrv, dbname = hh_config$database$name, user = hh_config$database$read_user, port = hh_config$database$port, host = hh_config$database$host, password = db_password))
+dbconstr <<-paste('dbname = \'', hh_config$database$name, '\', user = \'', hh_config$database$read_user, '\', port = ', hh_config$database$port, ', host = \'', hh_config$database$host, '\', password = \'', hh_config$database$read_pass,'\'', sep="")
+if (hh_debug) {
+  msg <- paste('logger -p user.notice dsccon: ', dbconstr)
+  system(msg)
+}
+
+if (class(dsccon) != "try-error") {
+	dbSendQuery(dsccon, "set statement_timeout to 300000;")
+	source(paste(hh_config$directories$web_source, "/R/prepareStatement.R", sep = ""))
+	source(paste(hh_config$directories$web_source, "/R/generatePlot.R", sep = ""))
+	switch(GET$type,
+		validateDBVersion  = {validateDbVersion()},
+		initPlotDDHtml     = {initPlotDDHtml()},
+		initServerDDHtml   = {initServerDDHtml()},
+		getDefaultServer   = {getDefaultServer()},
+		initNodeTabsHtml   = {initNodeTabsHtml()},
+		initNodeData       = {initNodeData()},
+		getDefaultPlotId   = {getDefaultPlotId()},
+		getDefaultGrouping = {getDefaultGrouping()},
+		getSupportURL      = {getSupportURL()},
+		generatePlot       = {generatePlot()},
+		generateYaml       = {generateYaml(dsccon)},
+		getEnableNodeSelection = {getEnableNodeSelection()}
+	)
+	dbDisconnect(dsccon)
+} else {
+	%>Error: Could not connect to the database - please check the connection parameters: name, read-user, port<%
+	return(FALSE)
+}
+%>
diff --git a/www/hedgehog/www/hedgehog.html.in b/www/hedgehog/www/hedgehog.html.in
new file mode 100644
index 0000000..c1cf0e0
--- /dev/null
+++ b/www/hedgehog/www/hedgehog.html.in
@@ -0,0 +1,168 @@
+<!--
+Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+
+This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0. If a copy of the MPL was not distributed with this
+file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+Developed by Sinodun IT (www.sinodun.com)
+-->
+
+<!DOCTYPE html>
+<!--[if lt IE 7 ]><html class="ie ie6" lang="en"> <![endif]-->
+<!--[if IE 7 ]><html class="ie ie7" lang="en"> <![endif]-->
+<!--[if IE 8 ]><html class="ie ie8" lang="en"> <![endif]-->
+<!--[if (gte IE 9)|!(IE)]><!--><html lang="en"> <!--<![endif]-->
+<head>
+
+	<!-- Basic Page Needs
+  ================================================== -->
+	<meta charset="utf-8">
+	<title>DNS-STATS:Hedgehog</title>
+	<meta name="Hedgehog" content="">
+	<meta name="jad@sinodun.com" content="">
+
+	<!-- Mobile Specific Metas
+  ================================================== -->
+	<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
+
+	<!-- CSS
+  ================================================== -->
+	<link rel="stylesheet" href="stylesheets/base.css?version=@HH_PACKAGE_VERSION@">
+	<link rel="stylesheet" href="stylesheets/skeleton.css?version=@HH_PACKAGE_VERSION@">
+	<link rel="stylesheet" href="stylesheets/layout.css?version=@HH_PACKAGE_VERSION@">
+
+	<!--[if lt IE 9]>
+		<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
+	<![endif]-->
+
+	<!-- Favicons
+   ================================================== -->
+	<link rel="shortcut icon" href="images/favicon.ico">
+	<link rel="apple-touch-icon" href="images/apple-touch-icon.png">
+	<link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
+	<link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
+        
+</head>
+<body>
+  <div class="container">
+    <hr style='margin-bottom:5px; margin-top:1px;'/>
+    <div class="three columns">
+      <div class="logo" id="logo">
+          <h5 class="remove-bottom" style="text-align: center; font-weight: bold">DNS-STATS</h5>
+          <h5 class="remove-bottom" style="text-align: center">Hedgehog @HH_PACKAGE_VERSION@:<br>
+          <a href="https://en.wikipedia.org/wiki/Hedgehog#Genera_and_species">@HH_VERSION_NAME@ release</a></h5>
+      </div>
+    </div>
+    <form method="get" name="plotParams">
+      <div class="four columns" id="timedisplay"><!-- 
+        auto populated by datetime.js
+   --></div>
+      <div class="five columns" id="timeTabs">
+        <div class="basicadv">
+          <a id="tfbasic" class="strong">Basic Time</a> | <a id="tfadv">Advanced time</a> <input type="button" id="setToday" value="Today" title="Set 'To' date equal to now and 'From' date equal to this time yesterday"> 
+        </div>
+        <div id="timeTabsBasic">
+          <div class="mycolumn" id="lefttimebtns"><!--
+         --><input type="button" id="lextend" value="+<" title="Extend time frame window by selected unit" ><!--
+         --><input type="button" id="larrow" value="<" title="Shift time frame window" ><!--
+       --></div>
+          <div class="mycolumn" id="tframebtns"><!--
+         --><input id="1h" type="radio" name="tframe" value="1"><!--
+         --><label class="button" for="1h">1 hr</label><!--
+         --><input id="4h" type="radio" name="tframe" value="4"><!--
+         --><label class="button" for="4h">4 hrs</label><!--
+         --><input id="12h" type="radio" name="tframe" value="12"><!--
+         --><label class="button" for="12h">12 hrs</label><!--
+         --><br /><!--
+         --><input id="day" type="radio" name="tframe" value="24"><!--
+         --><label class="button" for="day">day</label><!--
+         --><input id="wk" type="radio" name="tframe" value="168"><!--
+         --><label class="button" for="wk">week</label><!--
+         --><input id="mnth" type="radio" name="tframe" value="730"><!--
+         --><label class="button" for="mnth">month</label><!--
+       --></div>
+          <div class="mycolumn" id="righttimebtns"><!--
+         --><input type="button" id="rarrow" value=">" title="Shift time frame window" ><!--
+         --><input type="button" id="rextend" value=">+" title="Extend time frame window by selected unit" ><!--
+       --></div>
+        </div>
+        <div id="timeTabsAdv">
+          <div>
+            <label for="start" >Start</label>
+            <input type="datetime-local" name="start" id="start" />
+          </div>
+          <div>
+            <label for="stop" >End</label>
+            <input type="datetime-local" name="stop" id="stop" />
+          </div>
+        </div>
+      </div>
+      <div class="four columns" id="plotselection">
+        <div id="radioplt">
+            <input type="radio" name="rbplot" id="ggplot" title="Static PNG plot" checked>
+            <label for="ggplot" title="Static PNG plot">Static plot</label>
+            <input type="radio" name="rbplot" id="googleviz" title="Live Google Viz plot">
+            <label for="googleviz" title="Live Google Viz plot">Interactive plot</label>
+        </div>
+        <label for="plotType">Plot</label>
+        <select name="plotType" id="plotType"><!-- 
+        auto populated by gengraph.js
+   --></select>
+        <input id="generate" name="gobutton" type="button" value="Generate Plot!">  
+      </div>
+      <div class="sixteen columns" id="userdisplay"><!-- 
+        auto populated by datetime.js
+   --></div>   
+      <div class="sixteen columns" id="outerplot"><!-- 
+        auto populated by gengraph.js
+   --></div>
+      <hr style='margin-top:5px;margin-bottom:5px'/>
+      <div class="sixteen columms" id="srvr_select">
+      <label for="servers">Servers</label>
+      <select name="servers" id="servers" onchange="serverTab()">
+<!-- 
+        auto populated by gengraph.js
+   --></select>
+      </div>
+      <div class="sixteen columns" id="nodetabs"><!-- 
+         auto populated by nodetabs.js
+   --></div>
+    </form>
+    <div class="eight columns" id="local_links" >
+      <hr style='margin-bottom:5px;' />
+      Local Links: &nbsp; <a href="../rssac" target="_blank">RSSAC reports</a> &nbsp; <a href="about.html" target="_blank">Hedgehog QuickHelp</a>
+    </div>
+    <!-- TODO: Better job of links -->
+    <div class="eight columns" id="ext_links" >
+      <hr style='margin-bottom:5px;' />
+      External Links:  &nbsp; <a href="https://github.com/dns-stats/hedgehog/wiki" target="_blank">Hedgehog Wiki</a> &nbsp; <a href="http://dns-stats.org/" target="_blank">dns-stats.org</a>
+      &nbsp; <a id="support_link" href="https://github.com/dns-stats/hedgehog/issues" target="_blank">Support</a>
+    </div>
+    <div class="sixteen columns" id="copyright" >
+      <hr style='margin-top:5px;' />
+      &copy; <a href="http://www.icann.org/">Internet Corporation for Assigned Names and Numbers</a> <br/>Developed by <a href="http://sinodun.com/">Sinodun Internet Technologies</a>.
+    
+      <hr style='margin-top:5px;' />
+      This product includes GeoLite2 data created by MaxMind, available from
+      <a href="http://www.maxmind.com">http://www.maxmind.com</a>.
+	  <br/>Powered by: <a href="http://www.postgresql.org">PostgreSQL</a>, <a href="http://rapache.net/index.html">RApache</a>, <a href="https://httpd.apache.org">Apache</a>, 
+	  <a href="https://developers.google.com/chart/?hl=en">Google Charts</a>, <a href="https://www.r-project.org">R</a>, <a href="https://jquery.com">jQuery</a>, <a href="https://getdnsapi.net">getdns</a>
+    </div>  
+  </div><!-- container -->
+
+</body>
+
+    <!-- Javascript
+  ================================================== 
+    <script language="javascript" type="text/javascript"
+             src="/javascript/jquery/jquery.min.js"></script>
+    <script language="javascript" type="text/javascript"
+             src="/javascript/jquery-ui/jquery-ui.min.js"></script> -->
+    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
+    <script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js"></script>
+    <script src="js/gengraph.js?version=@HH_PACKAGE_VERSION@" type="text/javascript"></script>
+    <script src="js/datetimes.js?version=@HH_PACKAGE_VERSION@" type="text/javascript"></script>
+    <script src="js/nodetabs.js?version=@HH_PACKAGE_VERSION@" type="text/javascript"></script>
+
+</html>
diff --git a/www/hedgehog/www/images/ajax-loader.gif b/www/hedgehog/www/images/ajax-loader.gif
new file mode 100644
index 0000000..55f660e
Binary files /dev/null and b/www/hedgehog/www/images/ajax-loader.gif differ
diff --git a/www/hedgehog/www/images/all.png b/www/hedgehog/www/images/all.png
new file mode 100644
index 0000000..6691929
Binary files /dev/null and b/www/hedgehog/www/images/all.png differ
diff --git a/www/hedgehog/www/images/apple-touch-icon-114x114.png b/www/hedgehog/www/images/apple-touch-icon-114x114.png
new file mode 100644
index 0000000..53de2c4
Binary files /dev/null and b/www/hedgehog/www/images/apple-touch-icon-114x114.png differ
diff --git a/www/hedgehog/www/images/apple-touch-icon-72x72.png b/www/hedgehog/www/images/apple-touch-icon-72x72.png
new file mode 100644
index 0000000..c84037d
Binary files /dev/null and b/www/hedgehog/www/images/apple-touch-icon-72x72.png differ
diff --git a/www/hedgehog/www/images/apple-touch-icon.png b/www/hedgehog/www/images/apple-touch-icon.png
new file mode 100644
index 0000000..4ef016a
Binary files /dev/null and b/www/hedgehog/www/images/apple-touch-icon.png differ
diff --git a/www/hedgehog/www/images/calendar.png b/www/hedgehog/www/images/calendar.png
new file mode 100644
index 0000000..f5b6e7c
Binary files /dev/null and b/www/hedgehog/www/images/calendar.png differ
diff --git a/www/hedgehog/www/images/favicon.ico b/www/hedgehog/www/images/favicon.ico
new file mode 100644
index 0000000..9e3a702
Binary files /dev/null and b/www/hedgehog/www/images/favicon.ico differ
diff --git a/www/hedgehog/www/images/help.png b/www/hedgehog/www/images/help.png
new file mode 100644
index 0000000..db2141a
Binary files /dev/null and b/www/hedgehog/www/images/help.png differ
diff --git a/www/hedgehog/www/images/info.png b/www/hedgehog/www/images/info.png
new file mode 100644
index 0000000..e7fb9d5
Binary files /dev/null and b/www/hedgehog/www/images/info.png differ
diff --git a/www/hedgehog/www/images/none.png b/www/hedgehog/www/images/none.png
new file mode 100644
index 0000000..1dceffc
Binary files /dev/null and b/www/hedgehog/www/images/none.png differ
diff --git a/www/hedgehog/www/images/some.png b/www/hedgehog/www/images/some.png
new file mode 100644
index 0000000..a5486c9
Binary files /dev/null and b/www/hedgehog/www/images/some.png differ
diff --git a/www/hedgehog/www/js/datetimes.js b/www/hedgehog/www/js/datetimes.js
new file mode 100644
index 0000000..8387ef1
--- /dev/null
+++ b/www/hedgehog/www/js/datetimes.js
@@ -0,0 +1,237 @@
+///
+/// Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+///
+/// This Source Code Form is subject to the terms of the Mozilla Public
+/// License, v. 2.0. If a copy of the MPL was not distributed with this
+/// file, you can obtain one at https://mozilla.org/MPL/2.0/.
+///
+/// Developed by Sinodun IT (www.sinodun.com)
+///
+
+$(document).ready(function() {
+    // This function is called when hedgehog.html has loaded.
+    // Any other document.ready functions within included javascript
+    // files will also be called non-deterministically.
+    
+    // register callback function for when the advanced time input selection link is clicked
+    $("#tfadv").click(function() {
+        $("#timeTabsBasic").addClass("hidden");
+        $("#tfbasic").removeClass("strong");
+        $("#timeTabsAdv").removeClass("hidden");
+        $("#tfadv").addClass("strong");
+    });
+    
+    // register callback function for when the basic time input selection link is clicked
+    $("#tfbasic").click(function() {
+        $("#timeTabsAdv").addClass("hidden");
+        $("#tfadv").removeClass("strong");
+        $("#timeTabsBasic").removeClass("hidden");
+        $("#tfbasic").addClass("strong");
+    });
+    
+    // this function is no longer used, since we removed the debug option from the production release
+    $("#debug").click(function() {
+        $("#debugadv").toggleClass("strong");
+        $("#debug").toggleClass("strong");
+    });
+    
+    // register a callback function for when the Today button is clicked
+    $("#setToday").click(function set_today() {
+        window.stop = new Date();
+        window.stop.setUTCSeconds(0,0);
+        set_start_to_stop_dt();
+        set_start_to_stop_tm();
+        window.start.setUTCMinutes(window.start.getUTCMinutes() + 1);
+        getNewDateTime(window.start, -24);
+        setAdvInputs();
+        setUserDisplayMsg();
+        $('#day').prop('checked', true);
+    });
+    
+    // register a callback function for when the basic time control +< button is clicked
+    $("#lextend").click(function() {
+        extend_tframe("left");
+    });
+    
+    // register a callback function for when the basic time control >+ button is clicked
+    $("#rextend").click(function() {
+        extend_tframe("right");
+    });
+    
+    // register a callback function for when the basic time control < button is clicked
+    $("#larrow").click(function() {
+        shift_tframe("left");
+    });
+    
+    // register a callback function for when the basic time control > button is clicked
+    $("#rarrow").click(function() {
+        shift_tframe("right");
+    });
+    
+    // register a callback function for when any basic time control time frame buttons are clicked
+    $("input:radio[name='tframe']").change(function() {
+       set_tframe(parseInt($("input:radio[name='tframe']:checked").val()));
+    });
+    
+    // register a callback function for when the advanced time control start input is changed
+    $("#start").change(function() {
+        new_start_str = $('#start').val() + ":00Z"
+        if (window.start !== new_start_str) {
+            window.start = new Date(new_start_str);
+            setUserDisplayMsg();
+        }            
+    });
+    
+    // register a callback function for when the advanced time control stop input is changed
+    $("#stop").change(function() {
+        new_stop_str = $('#stop').val() + ":00Z"
+        if (window.stop !== new_stop_str) {
+            window.stop = new Date(new_stop_str);
+            setUserDisplayMsg();
+        }
+    });
+    
+});
+
+
+function setUserDisplayMsg() {
+    // update timedisplay and userdisplay messages
+    var tmsg = '<p class="remove-bottom"><span class="strong">From </span>' + window.start.toUTCString().replace('GMT','UTC') + '<br /><span class="strong">To </span>' + window.stop.toUTCString().replace('GMT','UTC') + '</p>';
+    $('#timedisplay').html(tmsg);
+    
+    enableGenerate(true);
+    
+    var existing = $('#userdisplay').html();
+    var invalidmsg = '<p class="error remove-bottom"><span class="strong">Error:</span> you have selected an End Date earlier than or equal to the Start date - please make a new selection</p>'; 
+    
+    if(window.stop <= window.start) {
+        enableGenerate(false);
+        if (existing.indexOf(invalidmsg) === -1) $('#userdisplay').html(existing + invalidmsg);
+    } else {
+        $('#userdisplay').html(existing.replace(invalidmsg, ""));
+    }
+    
+    var existing = $('#userdisplay').html();
+    var futuremsg = '<p class="warning remove-bottom"><span class="strong">Warning:</span> you have selected a future End Date - by default you will only see data up to the current moment</p>';
+    
+    if((window.stop > new Date()) || (window.start > new Date())){
+        if (existing.indexOf(futuremsg) === -1) $('#userdisplay').html(existing + futuremsg);
+    } else {
+        $('#userdisplay').html(existing.replace(futuremsg, ""));
+    }
+}
+
+function getNewDateTime(d, hours){
+    // add / subtract the number of hours from datetime, d,
+    // according to switch below and return new datetime
+    // default will return the same date (no default case)
+    switch(hours) {
+        case -730:
+            d.setMonth(d.getMonth() - 1); //d - 1 month
+            break;
+        case -168:
+            d.setDate(d.getDate() - 7); //d - 1 week
+            break;
+        case -24:
+            d.setDate(d.getDate() - 1); //d - 1 day
+            break;
+        case -12:
+            d.setHours(d.getHours() - 12); //d - 12 hours
+            break;
+        case -4:
+            d.setHours(d.getHours() - 4); //d - 4 hours
+            break;
+        case -1:
+            d.setHours(d.getHours() - 1); //d - 1 hour
+            break;
+        case 1:
+            d.setHours(d.getHours() + 1); //d + 1 hour
+            break;
+        case 4:
+            d.setHours(d.getHours() + 4); //d + 4 hours
+            break;
+        case 12:
+            d.setHours(d.getHours() + 12); //d + 12 hours
+            break;
+        case 24:
+            d.setDate(d.getDate() + 1); //d + 1 day
+            break;
+        case 168:
+            d.setDate(d.getDate() + 7); //d + 1 week
+            break;
+        case 730:
+            d.setMonth(d.getMonth() + 1); //d + 1 month
+            break;
+    }
+}
+
+function reset_hours() {
+    window.start.setUTCHours(0,0,0);
+    window.stop.setUTCHours(23,59,0);
+}
+
+function set_start_to_stop_dt() {
+    // sets start date equal to stop date
+    window.start.setUTCFullYear(window.stop.getUTCFullYear(), window.stop.getUTCMonth(), window.stop.getUTCDate());
+}
+
+function set_start_to_stop_tm() {
+    // sets start time equal to stop time
+    window.start.setUTCHours(window.stop.getUTCHours(), window.stop.getUTCMinutes(), 0, 0);
+}
+
+function set_tframe(hours) {
+    set_start_to_stop_dt();
+    set_start_to_stop_tm();
+    switch(hours) {
+        case 1:
+        case 4:
+        case 12:
+            window.start.setUTCHours(window.start.getUTCHours() - hours);
+            break;
+        case 24:
+            window.start.setUTCDate(window.start.getUTCDate() - 1);          
+            break;
+        case 168:
+            window.start.setUTCDate(window.start.getUTCDate() - 7);
+            break;            
+        case 730:
+            window.start.setUTCMonth(window.start.getUTCMonth() - 1);
+            break;
+        default: // do a day
+            window.start.setUTCDate(window.start.getUTCDate() + 1);          
+            break;
+    }
+    window.start.setUTCMinutes(window.start.getUTCMinutes() + 1);
+    setAdvInputs();
+    setUserDisplayMsg();    
+}
+
+function shift_tframe(direction) {
+    var hours = $("input[type='radio'][name='tframe']:checked").val() * 1; // current time frame increment selected
+    if (direction === 'left') {
+        hours = -hours;
+    }
+    getNewDateTime(window.start, hours);
+    getNewDateTime(window.stop, hours);
+    setAdvInputs();
+    setUserDisplayMsg();
+}
+
+function extend_tframe(direction) {
+    var hours = $("input[type='radio'][name='tframe']:checked").val() * 1; // current time frame increment selected
+    if (direction === 'left') {
+        getNewDateTime(window.start, -hours);
+    }else if (direction === 'right') {
+        getNewDateTime(window.stop, hours);
+    }
+    setAdvInputs();
+    setUserDisplayMsg(); 
+}
+
+function setAdvInputs() {
+    // sets the timeTabsAdv inputs
+    // slice removes the final ":###Z" which is not accepted by datetime input
+    $('#start').val(window.start.toISOString().slice(0,16));
+    $('#stop').val(window.stop.toISOString().slice(0,16));
+}
\ No newline at end of file
diff --git a/www/hedgehog/www/js/gengraph.js b/www/hedgehog/www/js/gengraph.js
new file mode 100755
index 0000000..d868d0b
--- /dev/null
+++ b/www/hedgehog/www/js/gengraph.js
@@ -0,0 +1,331 @@
+///
+/// Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+///
+/// This Source Code Form is subject to the terms of the Mozilla Public
+/// License, v. 2.0. If a copy of the MPL was not distributed with this
+/// file, you can obtain one at https://mozilla.org/MPL/2.0/.
+///
+/// Developed by Sinodun IT (www.sinodun.com)
+///
+
+$(document).ready(function() {
+    // This function is called when hedgehog.html has loaded.
+    // Any other document.ready functions within included javascript
+    // files will also be called non-deterministically.
+    
+    // hide advanced time controls
+    $("#timeTabsAdv").addClass("hidden");
+
+    // pre-select day from basic time handling
+    $('#day').prop('checked', true);
+    
+    // initialise start and stop datetimes
+    window.start = new Date();
+    window.start.setUTCDate(window.start.getUTCDate() -1);
+    window.stop = new Date(window.start);
+    reset_hours();
+    setAdvInputs();
+    setUserDisplayMsg();
+    
+    // initialise customMsg
+    window.customMsg = null;
+    
+    // initialise lastPlot and selectedOpt
+    window.lastPlot = null;
+    window.selectedOpt = null;
+
+    // populate plot and server dropdowns and nodetabs then generate default plot    
+    // , brew("initnodeslist"),
+    $.when(brew("validateDBVersion"),brew("initPltType"), brew("initPlotDDHtml"), brew("initServerDDHtml"), 
+           brew("getDefaultServer"), brew("initNodeData"), brew("getDefaultGrouping"), brew("getDefaultPlotId"), 
+           brew("getSupportURL"), brew("getEnableNodeSelection")).done(function(db,rp, pt, ss, ds, nd, dg, dp, su, ns){
+
+        if(db[0].indexOf("Error: Database version mismatch.") > -1) {
+            setDbVersionlMsg(true);
+            $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_connection.png" /><hr /></div>');
+            return;
+        }else if(db[0].indexOf("Error: Could not connect to the database") > -1) {
+            setDbFailMsg(true);
+            $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_connection.png" /><hr /></div>');
+            return;
+        }
+
+        //initialise plot radio selection
+        if (rp[0].indexOf('interactive') !== -1) {
+            $('#googleviz').prop('checked', true);
+        }
+
+        // initialise plot drop down
+        $("#plotType").html(pt[0]);
+        $("#plotType").val(parseInt(dp[0]));
+
+        // Initialise Server drop down item
+        $("#servers").html(ss[0]);
+
+        // check we have at least 1 server
+        var servers_ok = server_list_has_content();
+        if(!servers_ok) {
+            alert("Error: No servers are configured in the database");
+            $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_results.png" /><hr /></div>');
+            setNoResultsMsg(true);
+            return;
+        }
+
+        // checks user defined default for undefined, null and empty string
+        if (!(!ds[0] || ds[0].length === 0 || !ds[0].trim())) {
+            var options= document.getElementById('servers').options;
+            for (var i = 0; i < options.length; i++) {
+                if (options[i].text === ds[0].trim()) {
+                    options[i].selected = true;
+                    break;
+                }
+            }
+        }
+
+        // initialise node tabs and grouping
+        initNodeHtml(nd[0], ns[0]);
+        $('#ng_' + dg[0]).prop('checked', true);
+
+        setServersGroups();
+        initnodetabs();
+        serverTab();
+        
+        // checks for undefined, null and empty string
+        if (!(!su[0] || su[0].length === 0 || !su[0].trim())) {
+            $("#support_link").attr('href', su[0]);
+        }
+
+        // generate default plot
+        genDSCGraph();
+        setInitHelpMsg(true);
+
+    });
+    
+    // register callback function for when the static / interactive radio buttons are clicked
+    $("input[type='radio'][name='rbplot']").click(function() {
+        enableGenerate(true);
+    });
+    
+    // register callback function for when the plot drop down selection is changed
+    $("#plotType").change(function() {
+       enableGenerate(true);
+    });
+
+    // register callback function for when the generate plot button is clicked
+    $("#generate").click(function() {
+        genDSCGraph();
+    });
+
+});
+
+function enableGenerate(ind) {
+    // enable or disable the generate plot button
+    if(ind){
+        $('#generate').prop("disabled", false);
+        $('#generate').attr("title", "Select all the criteria for your plot, then click 'Generate Plot'");
+        setNoResultsMsg(false);
+        setInitHelpMsg(false);
+    }else{
+        $('#generate').prop("disabled", true);
+        $('#generate').attr("title", "Select new plot criteria to activate this button.");
+    }
+}
+
+function brew(divId) {
+    // call hedgehog.brew with type parameter set to the calling divId
+    return $.ajax({
+        url: "brew/hedgehog.brew",
+        data: { 'type': divId}
+    });
+}
+
+function genDSCGraph() {
+    // call hedgehog.brew to generate a plot
+    var gvis = 0;
+    if ($('#googleviz').prop('checked') === true) gvis = 1;
+
+    var pltid = $("#plotType option:selected").val();
+    var svrnm_tmp = $("#servers option:selected").text();
+    var svrnm = parse_name(svrnm_tmp);
+    var svrid = $("#servers option:selected").val();
+        
+    var ndset = {};
+    var allselected = true;
+    $("input[type='checkbox'].nodeselection").each(function(){
+        if( $(this).attr('id').split('_')[2] === svrnm) {
+            if( $(this).is(':checked') ) {
+                var node_id = $(this).attr('id').split("_")[0];
+                ndset[node_id] = true;
+            }else{
+                allselected = false;
+            }
+        }
+    });
+    var ndarr = "";
+    if (allselected){
+        ndarr = -1;
+    }else{
+        for(var node in ndset){
+            ndarr = ndarr + node + ",";
+        }
+        ndarr = ndarr.substring(0, ndarr.length - 1);
+        if(ndarr === '') {
+            window.alert("You must select at least one node")
+            return
+        }
+    }
+    
+    var request = $.ajax({
+        url: "brew/hedgehog.brew",
+        data: { 'type': "generatePlot",
+                'start': window.start.toISOString().slice(0,16),
+                'stop': window.stop.toISOString().slice(0,16),
+                'gvis': gvis,
+                'pltid': pltid,
+                'svrnm' : svrnm,
+                'svrid': svrid,
+                'ndarr': ndarr},
+        beforeSend: function() {
+                enableGenerate(false);
+                $('body').css('cursor', 'wait');
+                $("#outerplot").html('<hr /><img src="images/ajax-loader.gif" alt="loading..." /><hr />');
+        }
+    });
+            
+    request.done(function(data) {
+        $("#outerplot").html(data);
+        $('body').css('cursor', 'auto');
+        setUserMsg(null);
+        if (window.lastPlot === pltid){
+            if(window.selectedOpt !== null) {
+                if ( $(window.selectedOpt).prop('checked') === false ){
+                    $(window.selectedOpt).prop('checked', true);
+                    toggle_plot();
+                }
+                
+            }
+        }
+        window.lastPlot = pltid;
+    });
+    
+    request.fail(function(jqXHR, textStatus) {
+        $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_results.png" /><hr /></div>');
+        $('body').css('cursor', 'auto');
+        enableGenerate(true);
+        setUserMsg("Request failed with status '" + textStatus + "'");
+        window.lastPlot = null;
+    });
+}
+
+
+function toggle_plot() {  
+    // hide and display linear, stacked or log plots based on user selection
+    if ($("#rbliny").prop('checked')) {
+        window.selectedOpt = "#rbliny";
+        $('#liny').removeClass('hidden');
+        $('#logy').addClass('hidden');
+        $('#stack').addClass('hidden');
+        $("#ploturi").val($("#liny").prop("name"));
+    } else if ($("#rblogy").prop('checked')) {
+        window.selectedOpt = "#rblogy";
+        $('#liny').addClass('hidden');
+        $('#logy').removeClass('hidden');
+        $('#stack').addClass('hidden');
+        $("#ploturi").val($("#logy").prop("name"));
+    } else if ($("#rbstack").prop('checked')) {
+        window.selectedOpt = "#rbstack";
+        $('#liny').addClass('hidden');
+        $('#logy').addClass('hidden');
+        $('#stack').removeClass('hidden');
+        $("#ploturi").val($("#stack").prop("name"));
+    }
+}
+
+function setCustomMsg(msg) {
+    var existing = $('#userdisplay').html();
+    if (window.customMsg !== null) {
+        if (existing.indexOf(window.customMsg) !== -1) $('#userdisplay').html(existing.replace(window.customMsg, ""));
+        window.customMsg = null;
+    }
+    if (msg !== null) {
+        window.customMsg = '<p class="error remove-bottom"><span class="strong">Error:</span> ' + msg + '</p>';
+        if (existing.indexOf(window.customMsg) === -1) $('#userdisplay').html(existing + window.customMsg);
+    }
+}
+
+function setInitHelpMsg(ind) {
+    var existing = $('#userdisplay').html();
+    var msg = '<p class="info remove-bottom"><span class="strong">Info:</span> To generate a new plot, change the time, node or plot type and click \'Generate Plot.\'</p>';
+    if (ind) {
+        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
+    } else {
+        $('#userdisplay').html(existing.replace(msg, ""));
+    }
+}
+
+function setDbFailMsg(ind) {
+    var existing = $('#userdisplay').html();
+    var msg = '<p class="error remove-bottom"><span class="strong">Error:</span> How embarrassing! We have been unable to connect to the database. Please contact an administrator.</p>';
+    if (ind) {
+        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
+    } else {
+        $('#userdisplay').html(existing.replace(msg, ""));
+    }
+}
+
+function setDbVersionlMsg(ind) {
+    var existing = $('#userdisplay').html();
+    var msg = '<p class="error remove-bottom"><span class="strong">Error:</span> There is a database version mismatch between the GUI and the database. Please contact an administrator.</p>';
+    if (ind) {
+        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
+    } else {
+        $('#userdisplay').html(existing.replace(msg, ""));
+    }
+}
+
+function setNoResultsMsg(ind) {
+    var existing = $('#userdisplay').html();
+    var msg = '<p class="info remove-bottom"><span class="strong">Info:</span> Sorry, your query didn\'t return any results. Please try again with new selection criteria.</p>';
+    if (ind) {
+        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
+    } else {
+        $('#userdisplay').html(existing.replace(msg, ""));
+    }
+}
+
+function setPlotNotSupportedMsg(ind) {
+    var existing = $('#userdisplay').html();
+    var msg = '<p class="info remove-bottom"><span class="strong">Info:</span> Sorry, this plot is not yet supported. Please try again with a different plot.</p>';
+    if (ind) {
+        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
+    } else {
+        $('#userdisplay').html(existing.replace(msg, ""));
+    }
+}
+
+function setUserMsg(msg) {
+    setCustomMsg(msg);
+    if ($('#outerplot').html().indexOf("plots/no_graph.png") !== -1) {
+        setPlotNotSupportedMsg(true);
+    } else {
+        setPlotNotSupportedMsg(false);
+    }
+    if ($('#outerplot').html().indexOf("plots/no_results.png") !== -1) {
+        setNoResultsMsg(true);
+    } else {
+        setNoResultsMsg(false);
+    }
+    if ($('#outerplot').html().indexOf("plots/no_connection.png") !== -1) {
+        setDbFailMsg(true);
+    } else {
+        setDbFailMsg(false);
+    }
+}
+
+function server_list_has_content() {
+    if ($("#servers option:selected").text() == '') {
+        return false;
+    } else {
+        return true;
+    }
+}
diff --git a/www/hedgehog/www/js/nodetabs.js b/www/hedgehog/www/js/nodetabs.js
new file mode 100644
index 0000000..16d8dd9
--- /dev/null
+++ b/www/hedgehog/www/js/nodetabs.js
@@ -0,0 +1,369 @@
+///
+/// Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
+///
+/// This Source Code Form is subject to the terms of the Mozilla Public
+/// License, v. 2.0. If a copy of the MPL was not distributed with this
+/// file, you can obtain one at https://mozilla.org/MPL/2.0/.
+///
+/// Developed by Sinodun IT (www.sinodun.com)
+///
+
+var sortByText = function (a, b) {
+     return $.trim($(a).text()).localeCompare($.trim($(b).text()));
+}
+
+function parse_name( myid ) {
+    // Need to process the server name as it could contain full stops.
+    // Whilst these are allowed in identifiers if escaped their use complicates the code
+    // So instead replace the full stop with a different character.
+    // In CSS identifiers can contain only the characters [a-zA-Z0-9] and ISO 10646
+    // characters U+00A0 and higher, plus the hyphen (-) and the underscore (_). 
+    // We can't use with hyphen or underscore here, so use the Pound sign (U+00A3) which is valid.
+
+    return myid.replace( /\./g, "Â£" );
+}
+
+function initNodeHtml(nodes_raw, enable_node_selection) {
+
+    // Example data format
+    // var nodes = [
+    //   { server:     "Server-A",
+    //     groups:     [{group_name: "Region-1", node_list: [{node_name: "Node-1", node_id: "1"}, {node_name: "Node-2", node_id: "2", node_country: "US", node_city: "City-1", node_sg: "Subgroup-1"}]},
+    //                  {group_name: "Region-2", node_list: [{node_name: "Node-3", node_id: "3"}, {node_name: "Node-4", node_id: "4", node_country: "US", node_city: "City-1", node_sg: "Subgroup-2"}]}]},
+    //   { server:     "L-root",
+    //     groups:     [{group_name: "Europe",  node_list: [{node_name: "Node-5", node_id: "5"}, {node_name: "Node-6", node_id: "6", node_country: "US", node_city: "City-1", node_sg: "Subgroup-3"}}]},
+    //                  {group_name: "America", node_list: [{node_name: "Node-7", node_id: "7"}, {node_name: "Node-8", node_id: "8", node_country: "US", node_city: "City-1", node_sg: "Subgroup-3"}}]}]}];
+
+    nodes= jQuery.parseJSON(nodes_raw);
+
+    // Append requires a full element and will close any open elements so need care when using it.
+    $('#nodetabs').append("<div class='sixteen columns' id='groupcontent'></div>");
+    $('#nodetabs').append("<br class='clear'>");
+    $('#nodetabs').append("<div class='sixteen columns' id='nodecontent'>");
+    $('#srvr_select').append("<a style='font-weight:bold; font-size:80%; text-decoration: none; padding-left: 20px; padding-right: 5px;'>Group by:</a>");
+    $('#srvr_select').append("<input type='radio'          id='ng_country'  name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_country'  >Country </label>");
+    $('#srvr_select').append("<input type='radio'          id='ng_city'     name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_city'     >City </label>");
+    $('#srvr_select').append("<input type='radio'          id='ng_instance' name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_instance' >Instance </label>");
+    if (enable_node_selection == 1) {$('#srvr_select').append("<input type='radio'          id='ng_none'     name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_none'     >Node </label>");}
+
+    // Check for no servers is made earlier, so just check for nodes here
+    // if (nodes_raw = "[]") {
+    //     alert("No nodes found in the database");
+    // }
+
+    //*** SERVERS ***//
+    // For each server, construct the groups.
+    for (var i = 0; i < nodes.length; i++) {
+        var server_tmp = nodes[i].server;
+        var server     = parse_name(server_tmp);
+        $('#groupcontent'       ).append(  "<div class='hidden' id='" + server + "'>");
+        $('#' + server          ).append(  "<div class='Tabs'   id='" + server + "_tabs'>");
+        $('#' + server + '_tabs').append(  "<ul                 id='" + server +"_tabs_ul'>");
+        //***  GROUPS (REGIONS) ***//
+        // For each group, add a tab and fill in the nodes
+        for (var j = 0; j < nodes[i].groups.length; j++) {
+            var group_tmp = nodes[i].groups[j].group_name;
+            var group     = parse_name(group_tmp);
+            var group_server       = group + "_" + server;
+            var group_server_basic = group_server;
+            // Create the top tab with a list entry
+            $('#' + server + '_tabs_ul').append("<li id='li_" + group_server + "' onclick='gpTab(\"" + group_server + "\")'>  <a>" + group_tmp + "<img id='cb_" + group_server +"_img' src='images/all.png' alt='all selected' height='10' width='10'></a></li>");
+            //Now the node content divs
+            for (var node_grouping_options = 0; node_grouping_options <=3; node_grouping_options++) {
+                if (node_grouping_options == 1)      {group_server = group + "_" + server + "_subgroup";}
+                else if (node_grouping_options == 2) {group_server = group + "_" + server + "_city";}
+                else if (node_grouping_options == 3) {group_server = group + "_" + server + "_country";}
+                $('#nodecontent'                 ).append("<div class='group_showing'   id='" + group_server + "'>");
+                $('#' + group_server             ).append("<div class='allnone'         id='" + group_server + "_allnone'>");
+                $('#' + group_server             ).append("<div class='node_cbs'        id='" + group_server + "_node_cbs_id'>");
+                // Selection buttons
+                $('#' + group_server + '_allnone').append("<a style='font-weight:bold;'>Actions:</a>");
+                if (j == -1) {
+                    $('#' + group_server + '_allnone').append("<input type='button' id='selectAllBtn2'  value='Select all nodes'    title='Select all available nodes for plotting' onclick='selectAll(\"cb_" + group_server_basic + "\")'>"); 
+                    $('#' + group_server + '_allnone').append("<input type='button' id='selectNoneBtn2' value='De-select all nodes' title='De-select all available nodes'           onclick='selectNone(\"cb_" + group_server_basic + "\")'><hr>");
+                } else {
+                    $('#' + group_server + '_allnone').append("<input type='button' id='selectOnlyRegionBtn' value='Select only " + group + "' title='Select only the nodes in this region for plotting'   onclick='selectOnly(\"cb_" + group_server_basic + "\")'> ");
+                    $('#' + group_server + '_allnone').append("<input type='button' id='selectAllRegionBtn'  value='Include all " + group + "' title='Include all the nodes in this region for plotting'   onclick='selectAll(\"cb_"  + group_server_basic + "\")'> ");
+                    $('#' + group_server + '_allnone').append("<input type='button' id='selectNoneRegionBtn' value='Exclude all " + group + "' title='De-select all the nodes in this region for plotting' onclick='selectNone(\"cb_" + group_server_basic + "\")'><hr> ");
+                }
+                //*** NODES AND GROUPING OPTIONS***//
+                for (var k = 0; k < nodes[i].groups[j].node_list.length; k++) {
+                    var node                 = nodes[i].groups[j].node_list[k]
+                    var node_id_group_server = node.node_id + "_" + group_server_basic;
+                    if (node_grouping_options == 0) {
+                        $('#' + group_server + '_node_cbs_id').append("<input type='checkbox' class='nodeselection' id='" + node_id_group_server + "' name='cb_" + group_server + "' onclick='selectNode(\"" + node_id_group_server + "\")'>");
+                        $('#' + group_server + '_node_cbs_id').append("<label for='" + node_id_group_server + "' title='Toggle node selection for " + node.node_name + "\n Country: " + node.node_country + "\n City: " + node.node_city + "\n Instance: " + node.node_sg + "'>" + node.node_name + "</label>");
+                    } else {
+                        if (node_grouping_options == 1) {
+                            var node_groupby_label   = node.node_sg;
+                            var node_groupby_class   = 'subgroupselection';
+                        }
+                        if (node_grouping_options == 2) {
+                            var node_groupby_label   = node.node_city;
+                            var node_groupby_class   = 'cityselection';
+                        }   
+                        if (node_grouping_options == 3) {
+                            var node_groupby_label   = node.node_country;
+                            var node_groupby_class   = 'countryselection';
+                        }
+                        // use tilde not underscores as delimiter so other code doesn't trigger off this
+                        var node_groupby_group_server = node_groupby_label + "~" + group + "~" + server;
+                        if($("input[type='checkbox'][id='" + node_groupby_group_server + "']").length == 0) {
+                            $('#' + group_server + '_node_cbs_id').append("<label for='" + node_groupby_group_server + "' title='Toggle node selection for:\n" + node.node_name + "'>" + node_groupby_label + " <img id='cb_" + node_groupby_group_server +"_img' src='images/all.png' alt='all selected' height='10' width='10'>");
+                            $('#' + group_server + '_node_cbs_id').append("<input type='checkbox' class='" + node_groupby_class + "' id='" + node_groupby_group_server + "' name='cb-" + node_groupby_group_server + "' value='" + node_id_group_server + "' onclick='selectGrouping(this)'></label>");
+                        } else {
+                            var my_label = $("label[for='" + node_groupby_group_server + "']").attr("title");
+                            $("label[for='" + node_groupby_group_server + "']").attr("title", my_label + "\n" + node.node_name);
+                            var my_cb = $("input[type='checkbox'][id='" + node_groupby_group_server + "']");
+                            $.each(my_cb, function() {
+                               this.value = this.value + "~" + node_id_group_server; 
+                            });
+                        }
+                    }
+                }
+                if (node_grouping_options != 0) {
+                    // We must sort the country, city and instances alphabetically
+                    var sorted = $('#' + group_server + '_node_cbs_id label').sort(sortByText);
+                    $('#' + group_server + '_node_cbs_id').append(sorted);
+                }
+            }
+        }
+    }
+}
+
+function ngChanged() {
+    var active_group = $( "li.activeGpTab" );
+    $.each(active_group, function() {
+        var group_name = this.id.substring(3, this.id.length);
+        gpTab(group_name);
+    });
+}
+
+function setServersGroups() {
+    // sets window.servers to the set of server names,
+    // and window.groups to the set of group names.
+    // nodeselection checkbox name format: 'node-id_group_server'
+    window.servers = {};
+    window.groups = {};
+    $("input[type='checkbox'].nodeselection").each(function(){
+        var s = $(this).attr('name').split('_')[2];
+        var g = $(this).attr('name').split('_')[1];
+        window.servers[s] = true;
+        window.groups[g + "_" + s] = true;
+    });
+}
+
+function selectSisterGrouping(grp_class) {
+    $("input[type='checkbox']." + grp_class).each(function(){
+         node_list = this.value.split("~");
+         var all = true;
+         var none = true;
+         $.each(node_list, function(i, value){
+             var node_cb = $("#" + value);
+             if($(node_cb).is(':checked')){
+                 none = false;
+             }else{
+                 all = false;
+             }
+         });
+         if (all == true) {
+             $(this).prop('checked', true);
+             $("label[for='" + this.id + "']").css("color", '#000');
+             document.getElementById("cb_" + this.id + "_img").src="images/all.png";
+             document.getElementById("cb_" + this.id + "_img").alt="all selected";
+         } else if (none == true) {
+             $(this).prop('checked', false);
+             $("label[for='" + this.id + "']").css("color", '#999');
+             document.getElementById("cb_" + this.id + "_img").src="images/none.png";
+             document.getElementById("cb_" + this.id + "_img").alt="none selected";
+         } else {
+             $(this).prop('checked', true);
+             //$("label[for='" + this.id + "']").css("color", '#0072B2');
+             $("label[for='" + this.id + "']").css("color", '#000');
+             var temp = $("#cb_" + this.id + "_img");
+             document.getElementById("cb_" + this.id + "_img").src="images/some.png";
+             document.getElementById("cb_" + this.id + "_img").alt="some selected";
+         }
+     });
+}
+
+
+function selectSister(idList, ckd) {
+    // finds all checkboxes where their node id
+    // is in idList and sets their checked status
+    // to ckd (true / false)
+    $.each(idList, function(i, value){
+        var ids = value.split('_');
+        var nd = ids[0];
+        var gp = ids[1];
+        var srvr = ids[2];
+        for(gs in window.groups){
+            g = gs.split('_')[0];
+            s = gs.split('_')[1];
+            if((s === srvr) /* && (g !== gp)*/){
+                var sisterId = nd + '_' + g + '_' + srvr;
+                var sister = $("#" + sisterId);
+                if(sister){
+                    sister.prop('checked', ckd);
+                }
+            }
+        }
+    });
+    
+    // update to also set the cbs in the grouping tabs
+    var grouping = ["subgroupselection", "cityselection", "countryselection"]; 
+    for (i = 0; i < 3; i++) { 
+        selectSisterGrouping(grouping[i]);
+    }
+
+    // set group tab images to reflect
+    // whether all, some or no checkboxes
+    // are selected
+    initnodetabs();
+}
+
+function select(cbGp, ckd) {
+    // for all checkboxes with the name cbGp
+    // set checked status to ckd (true/false)
+    var idList = new Array();
+    $("input[type='checkbox'][name=" + cbGp + "]").each(function(){
+        $(this).prop('checked', ckd);
+        idList.push($(this).attr('id'));
+    });
+    // ensure any sister nodes are
+    // updated to maintain consistency
+    selectSister(idList, ckd);
+}
+
+function updateAll(ckd) {
+    // for all checkboxes 
+    // set checked status to ckd (true/false)
+    var idList = new Array();
+    $("input[type='checkbox']").each(function(){
+        $(this).prop('checked', ckd);
+        idList.push($(this).attr('id'));
+    });
+    // ensure any sister nodes are
+    // updated to maintain consistency
+    selectSister(idList, ckd);
+}
+
+function selectAll(cbGp) {
+    // set all checkboxes with the
+    // name cbGp to checked: true
+    select(cbGp,true);
+}
+
+function selectNone(cbGp) {
+    // set all checkboxes with the
+    // name cdGp to checked: false
+    select(cbGp,false);
+}
+
+function selectOnly(cbGp) {
+    // set all checkboxes to false then 
+	// set all with
+    // name cdGp to checked: true
+	updateAll(false)
+    select(cbGp,true);
+}
+
+
+function selectNode(ckbxid) {
+    // ensure any sister nodes are
+    // updated to maintain consistency
+    idList = new Array();
+    idList.push(ckbxid);
+    selectSister(idList, $("#" + ckbxid).is(':checked'));
+}
+
+function selectGrouping(item) {
+    idList = item.value.split("~");
+    selectSister(idList, $(item).is(':checked'));
+    var my_subgroup = item.id.split("~")[0];
+    var my_server   = item.id.split("~")[2];
+    $("input[type='checkbox']." + item.class).each(function(){
+        var this_subgroup = this.id.split("~")[0];
+        var this_server = this.id.split("~")[2];
+        if (my_server == this_server && my_subgroup == this_subgroup) {
+            this.checked = $(item).is(':checked');
+        }
+    });
+}
+
+function selectSome(cbGs) {
+    // determine whether all, some or no
+    // checkboxes are checked where their
+    // name is cbGs and then update the
+    // group tab image accordingly
+    var none = true;
+    var all = true;
+    
+    $("input[type='checkbox'][name=" + cbGs + "]").each(function() {
+        if($(this).is(':checked')){
+            none = false;
+        }else{
+            all = false;
+        }
+    });
+    
+    if(none){
+        $("#" + cbGs + "_img").attr("src", "images/none.png");
+        $("#" + cbGs + "_img").attr("alt", "none selected"); 
+    }else if(all){
+        $("#" + cbGs + "_img").attr("src", "images/all.png");
+        $("#" + cbGs + "_img").attr("alt", "all selected");
+    }else{
+        $("#" + cbGs + "_img").attr("src", "images/some.png");
+        $("#" + cbGs + "_img").attr("alt", "some selected");
+    }
+    
+}
+
+function initnodetabs(){
+    // for each group tab, set the image
+    // which indicates whether all, some
+    // or no checkboxes are selected
+
+    for(var gs in window.groups) {
+        selectSome('cb_' + gs);
+    }
+    enableGenerate(true);
+}
+
+function gpTab(gpnm) {
+    // set active group tab, display associated
+    // node content and hide old node content from view
+    $("li.activeGpTab").removeClass("activeGpTab");
+    $("#li_" + gpnm).addClass("activeGpTab");
+    old_content = $(".group_showing");
+    old_content.removeClass("group_showing");
+    old_content.addClass("hidden");
+    if ($('#ng_instance').prop('checked') === true) gpnm = gpnm + "_subgroup";
+    if ($('#ng_city').prop('checked')     === true) gpnm = gpnm + "_city";
+    if ($('#ng_country').prop('checked')  === true) gpnm = gpnm + "_country";
+    $("#" + gpnm).removeClass("hidden");
+    $("#" + gpnm).addClass("group_showing");
+}
+
+function serverTab() {
+    // display associated group tabs for selected
+    // server and hide old group tabs from view
+    gptabsid_tmp = $("#servers option:selected").text();
+    gptabsid = parse_name(gptabsid_tmp);
+    old_content = $("div.server_showing");
+    old_content.removeClass('server_showing');
+    old_content.addClass('hidden');
+    $("#" + gptabsid).removeClass('hidden');
+    $("#" + gptabsid).addClass('server_showing');
+    gpTab('All_' +  gptabsid);
+    for(var gs in window.groups) {
+        s = gs.split('_')[1];
+        if(s === gptabsid){
+            selectAll('cb_' + gs);
+        }else{
+            selectNone('cb_' + gs);
+        }
+        selectSome('cb_' + gs);
+    }
+}
\ No newline at end of file
diff --git a/www/hedgehog/www/plots/no_connection.png b/www/hedgehog/www/plots/no_connection.png
new file mode 100644
index 0000000..d236381
Binary files /dev/null and b/www/hedgehog/www/plots/no_connection.png differ
diff --git a/www/hedgehog/www/plots/no_graph.png b/www/hedgehog/www/plots/no_graph.png
new file mode 100644
index 0000000..b97ca4b
Binary files /dev/null and b/www/hedgehog/www/plots/no_graph.png differ
diff --git a/www/hedgehog/www/plots/no_results.png b/www/hedgehog/www/plots/no_results.png
new file mode 100644
index 0000000..df1a802
Binary files /dev/null and b/www/hedgehog/www/plots/no_results.png differ
diff --git a/www/hedgehog/www/stylesheets/base.css b/www/hedgehog/www/stylesheets/base.css
new file mode 100644
index 0000000..7cb892e
--- /dev/null
+++ b/www/hedgehog/www/stylesheets/base.css
@@ -0,0 +1,277 @@
+/*
+* Skeleton V1.2
+* Copyright 2011, Dave Gamache
+* www.getskeleton.com
+* Free to use under the MIT license.
+* http://www.opensource.org/licenses/mit-license.php
+* 6/20/2012
+*/
+
+
+/* Table of Content
+==================================================
+	#Reset & Basics
+	#Basic Styles
+	#Site Styles
+	#Typography
+	#Links
+	#Lists
+	#Images
+	#Buttons
+	#Forms
+	#Misc */
+
+
+/* #Reset & Basics (Inspired by E. Meyers)
+================================================== */
+	html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video {
+		margin: 0;
+		padding: 0;
+		border: 0;
+		font-size: 100%;
+		font: inherit;
+		vertical-align: baseline; }
+	article, aside, details, figcaption, figure, footer, header, hgroup, menu, nav, section {
+		display: block; }
+	body {
+		line-height: 1; }
+	ol, ul {
+		list-style: none; }
+	blockquote, q {
+		quotes: none; }
+	blockquote:before, blockquote:after,
+	q:before, q:after {
+		content: '';
+		content: none; }
+	table {
+		border-collapse: collapse;
+		border-spacing: 0; }
+
+
+/* #Basic Styles
+================================================== */
+	body {
+		background: #fff;
+		font: 14px/21px "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif;
+		color: #444;
+		-webkit-font-smoothing: antialiased; /* Fix for webkit rendering */
+		-webkit-text-size-adjust: 100%;
+ }
+
+
+/* #Typography
+================================================== */
+	h1, h2, h3, h4, h5, h6 {
+		color: #181818;
+		font-family: "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif;
+		font-weight: normal; }
+	h1 a, h2 a, h3 a, h4 a, h5 a, h6 a { font-weight: inherit; }
+	h1 { font-size: 46px; line-height: 50px; margin-bottom: 14px;}
+	h2 { font-size: 35px; line-height: 40px; margin-bottom: 10px; }
+	h3 { font-size: 28px; line-height: 34px; margin-bottom: 8px; }
+	h4 { font-size: 21px; line-height: 30px; margin-bottom: 4px; }
+	h5 { font-size: 17px; line-height: 24px; }
+	h6 { font-size: 14px; line-height: 21px; }
+	.subheader { color: #777; }
+
+	p { margin: 0 0 20px 0; }
+	p img { margin: 0; }
+	p.lead { font-size: 21px; line-height: 27px; color: #777;  }
+
+	em { font-style: italic; }
+	strong { font-weight: bold; color: #333; }
+	small { font-size: 80%; }
+
+/*	Blockquotes  */
+	blockquote, blockquote p { font-size: 17px; line-height: 24px; color: #777; font-style: italic; }
+	blockquote { margin: 0 0 20px; padding: 9px 20px 0 19px; border-left: 1px solid #ddd; }
+	blockquote cite { display: block; font-size: 12px; color: #555; }
+	blockquote cite:before { content: "\2014 \0020"; }
+	blockquote cite a, blockquote cite a:visited, blockquote cite a:visited { color: #555; }
+
+	hr { border: solid #ddd; border-width: 1px 0 0; clear: both; margin: 10px 0 30px; height: 0; }
+
+
+/* #Links
+================================================== */
+	a, a:visited { color: #333; text-decoration: underline; outline: 0; }
+	a:hover, a:focus { color: #000; }
+	p a, p a:visited { line-height: inherit; }
+
+
+/* #Lists
+================================================== */
+	ul, ol { margin-bottom: 20px; }
+	ul { list-style: none outside; }
+	ol { list-style: decimal; }
+	ol, ul.square, ul.circle, ul.disc { margin-left: 30px; }
+	ul.square { list-style: square outside; }
+	ul.circle { list-style: circle outside; }
+	ul.disc { list-style: disc outside; }
+	ul ul, ul ol,
+	ol ol, ol ul { margin: 4px 0 5px 30px; font-size: 90%;  }
+	ul ul li, ul ol li,
+	ol ol li, ol ul li { margin-bottom: 6px; }
+	li { line-height: 18px; margin-bottom: 12px; }
+	ul.large li { line-height: 21px; }
+	li p { line-height: 21px; }
+
+/* #Images
+================================================== */
+
+	img.scale-with-grid {
+		max-width: 100%;
+		height: auto; }
+
+
+/* #Buttons
+================================================== */
+
+	.button,
+	button,
+	input[type="submit"],
+	input[type="reset"],
+	input[type="text"],
+	input[type="button"] {
+		background: #eee; /* Old browsers */
+		background: #eee -moz-linear-gradient(top, rgba(255,255,255,.2) 0%, rgba(0,0,0,.2) 100%); /* FF3.6+ */
+		background: #eee -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.2)), color-stop(100%,rgba(0,0,0,.2))); /* Chrome,Safari4+ */
+		background: #eee -webkit-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Chrome10+,Safari5.1+ */
+		background: #eee -o-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Opera11.10+ */
+		background: #eee -ms-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* IE10+ */
+		background: #eee linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* W3C */
+	  border: 1px solid #aaa;
+	  border-top: 1px solid #ccc;
+	  border-left: 1px solid #ccc;
+	  -moz-border-radius: 3px;
+	  -webkit-border-radius: 3px;
+	  border-radius: 3px;
+	  color: #444;
+	  display: inline-block;
+	  font-size: 11px;
+	  font-weight: bold;
+	  text-decoration: none;
+	  text-shadow: 0 1px rgba(255, 255, 255, .75);
+	  cursor: pointer;
+	  margin-bottom: 20px;
+	  line-height: normal;
+	  padding: 8px 10px;
+	  font-family: "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif; }
+
+	.button:hover,
+	button:hover,
+	input[type="submit"]:hover,
+	input[type="reset"]:hover,
+	input[type="button"]:hover {
+		color: #222;
+		background: #ddd; /* Old browsers */
+		background: #ddd -moz-linear-gradient(top, rgba(255,255,255,.3) 0%, rgba(0,0,0,.3) 100%); /* FF3.6+ */
+		background: #ddd -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.3)), color-stop(100%,rgba(0,0,0,.3))); /* Chrome,Safari4+ */
+		background: #ddd -webkit-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Chrome10+,Safari5.1+ */
+		background: #ddd -o-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Opera11.10+ */
+		background: #ddd -ms-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* IE10+ */
+		background: #ddd linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* W3C */
+	  border: 1px solid #888;
+	  border-top: 1px solid #aaa;
+	  border-left: 1px solid #aaa; }
+
+	.button:active,
+	button:active,
+	input[type="submit"]:active,
+	input[type="reset"]:active,
+	input[type="button"]:active {
+		border: 1px solid #666;
+		background: #ccc; /* Old browsers */
+		background: #ccc -moz-linear-gradient(top, rgba(255,255,255,.35) 0%, rgba(10,10,10,.4) 100%); /* FF3.6+ */
+		background: #ccc -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.35)), color-stop(100%,rgba(10,10,10,.4))); /* Chrome,Safari4+ */
+		background: #ccc -webkit-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Chrome10+,Safari5.1+ */
+		background: #ccc -o-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Opera11.10+ */
+		background: #ccc -ms-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* IE10+ */
+		background: #ccc linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* W3C */ }
+
+	.button.full-width,
+	button.full-width,
+	input[type="submit"].full-width,
+	input[type="reset"].full-width,
+	input[type="button"].full-width {
+		width: 100%;
+		padding-left: 0 !important;
+		padding-right: 0 !important;
+		text-align: center; }
+
+	/* Fix for odd Mozilla border & padding issues */
+	button::-moz-focus-inner,
+	input::-moz-focus-inner {
+    border: 0;
+    padding: 0;
+	}
+
+
+/* #Forms
+================================================== */
+
+	form {
+		margin-bottom: 20px; }
+	fieldset {
+		margin-bottom: 20px; }
+	input[type="text"],
+	input[type="password"],
+	input[type="email"],
+        input[type="date"],
+        input[type="datetime-local"],
+        input[type="time"],
+	textarea,
+	select {
+		border: 1px solid #ccc;
+		padding: 6px 4px;
+		outline: none;
+		-moz-border-radius: 2px;
+		-webkit-border-radius: 2px;
+		border-radius: 2px;
+		font: 13px "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif;
+		color: #777;
+		margin: 0;
+		width: 210px;
+		max-width: 100%;
+		display: inline;
+		margin-bottom: 20px;
+		background: #fff; }
+	select {
+		padding: 0; }
+	
+	input[type="text"]:focus,
+	input[type="password"]:focus,
+	input[type="email"]:focus,
+        input[type="date"]:focus,
+        input[type="datetime-local"]:focus,
+        input[type="time"]:focus,
+	textarea:focus {
+		border: 1px solid #aaa;
+ 		color: #444;
+ 		-moz-box-shadow: 0 0 3px rgba(0,0,0,.2);
+		-webkit-box-shadow: 0 0 3px rgba(0,0,0,.2);
+		box-shadow:  0 0 3px rgba(0,0,0,.2); }
+	textarea {
+		min-height: 60px; }
+	label,
+	legend {
+		display: inline;
+		font-weight: bold;
+		font-size: 13px;  }
+	select{
+		width: 220px; }
+	input[type="checkbox"] {
+		display: inline; }
+	label span,
+	legend span {
+		font-weight: normal;
+		font-size: 13px;
+		color: #444; }
+
+/* #Misc
+================================================== */
+	.remove-bottom { margin-bottom: 0 !important; }
+	.half-bottom { margin-bottom: 10px !important; }
+	.add-bottom { margin-bottom: 20px !important; }
+
+
diff --git a/www/hedgehog/www/stylesheets/layout.css b/www/hedgehog/www/stylesheets/layout.css
new file mode 100644
index 0000000..d2c62c4
--- /dev/null
+++ b/www/hedgehog/www/stylesheets/layout.css
@@ -0,0 +1,495 @@
+/*
+* Skeleton V1.2
+* Copyright 2011, Dave Gamache
+* www.getskeleton.com
+* Free to use under the MIT license.
+* http://www.opensource.org/licenses/mit-license.php
+* 6/20/2012
+*/
+
+/* Table of Content
+==================================================
+	#Site Styles
+	#Page Styles
+	#Media Queries
+	#Font-Face */
+
+/* #Site Styles
+================================================== */
+
+.strong {
+    font-weight: bold;
+}
+
+.logo {
+    #padding: 2px 2px;
+    color: #fff;
+    background-color: #fff;
+    border: 3px solid #5894b0;
+    -webkit-border-radius: 4px;
+    -moz-border-radius: 4px;
+    border-radius: 4px;    
+}
+
+.info {
+    padding: 2px 8px;
+    color: #3a87ad;
+    background-color: #d9edf7;
+    border: 1px solid #bce8f1;
+    -webkit-border-radius: 4px;
+    -moz-border-radius: 4px;
+    border-radius: 4px;    
+}
+
+.warning {
+    padding: 2px 8px;
+    color: #c09853;
+    background-color: #fcf8e3;
+    border: 1px solid #fbeed5;
+    -webkit-border-radius: 4px;
+    -moz-border-radius: 4px;
+    border-radius: 4px;
+}
+
+.error {
+    padding: 2px 8px;
+    color: #b94a48;
+    background-color: #f2dede;
+    border: 1px solid #eed3d7;
+    -webkit-border-radius: 4px;
+    -moz-border-radius: 4px;
+    border-radius: 4px;
+}
+
+.hidden {
+    display: none !important;
+}
+
+/*input, textarea {
+  /* stop iOS mobile safari from overriding my border radius settings */
+  /*-webkit-appearance: none;*/
+  -webkit-border-radius: 0;
+}*/
+
+ul.discs {
+    list-style-type: disc;
+}
+
+ul.circles {
+    list-style-type: circle;
+}
+
+/* #Page Styles
+================================================== */
+
+#helpimg {
+    position: relative;
+    top: 2px;
+    left: 8px;
+}
+
+#generate:not([disabled]){
+    background: #5894b0; /* Old browsers */
+    background: #5894b0 -moz-linear-gradient(top, rgba(255,255,255,.2) 0%, rgba(0,0,0,.2) 100%); /* FF3.6+ */
+    background: #5894b0 -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.2)), color-stop(100%,rgba(0,0,0,.2))); /* Chrome,Safari4+ */
+    background: #5894b0 -webkit-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Chrome10+,Safari5.1+ */
+    background: #5894b0 -o-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Opera11.10+ */
+    background: #5894b0 -ms-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* IE10+ */
+    background: #5894b0 linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* W3C */
+    color: white;
+    text-shadow: none;
+}
+
+#generate:not([disabled]):hover {
+    background: #4e8aa6; /* Old browsers */
+    background: #4e8aa6 -moz-linear-gradient(top, rgba(255,255,255,.3) 0%, rgba(0,0,0,.3) 100%); /* FF3.6+ */
+    background: #4e8aa6 -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.3)), color-stop(100%,rgba(0,0,0,.3))); /* Chrome,Safari4+ */
+    background: #4e8aa6 -webkit-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Chrome10+,Safari5.1+ */
+    background: #4e8aa6 -o-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Opera11.10+ */
+    background: #4e8aa6 -ms-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* IE10+ */
+    background: #4e8aa6 linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* W3C */
+}
+
+#generate:not([disabled]):active {
+    background: #42748b -moz-linear-gradient(top, rgba(255,255,255,.35) 0%, rgba(10,10,10,.4) 100%); /* FF3.6+ */
+    background: #42748b -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.35)), color-stop(100%,rgba(10,10,10,.4))); /* Chrome,Safari4+ */
+    background: #42748b -webkit-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Chrome10+,Safari5.1+ */
+    background: #42748b -o-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Opera11.10+ */
+    background: #42748b -ms-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* IE10+ */
+    background: #42748b linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* W3C */
+}
+
+input[type="button"][disabled] {
+    color: GrayText;
+    cursor: inherit;
+}
+
+input[type="button"][disabled]:hover {
+    color: GrayText;
+    cursor: inherit;
+    background: #eee; /* Old browsers */
+    background: #eee -moz-linear-gradient(top, rgba(255,255,255,.2) 0%, rgba(0,0,0,.2) 100%); /* FF3.6+ */
+    background: #eee -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.2)), color-stop(100%,rgba(0,0,0,.2))); /* Chrome,Safari4+ */
+    background: #eee -webkit-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Chrome10+,Safari5.1+ */
+    background: #eee -o-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Opera11.10+ */
+    background: #eee -ms-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* IE10+ */
+    background: #eee linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* W3C */
+    border: 1px solid #aaa;
+    border-top: 1px solid #ccc;
+    border-left: 1px solid #ccc;
+}
+
+#timeTabsAdv {
+    padding-left: 14px;
+    margin-bottom: 8px;
+}
+
+#timeTabsAdv label,
+#srvr_select label,
+#plotselection label,
+#start,
+#stop,
+#servers,
+#plotType {
+    display: inline-block;
+    margin-bottom: 0;
+    font-size: 80%;
+}
+
+#start,
+#stop,
+#plotType {
+    width: 176px;
+    padding: 0;
+}
+
+#setToday {
+    margin: 0 0 6px 32px;
+    padding: 2px 8px;
+}
+
+#plotType {
+    margin: 5px 0 0 0;
+}
+
+#generate {
+    margin: 4px 0 0 0;
+    padding: 2px 4px;
+    font-size: 80%;
+    width: 200px;
+}
+
+
+#timeTabsAdv label {
+    width: 26px;
+}
+
+#timeTabsAdv label,
+#srvr_select label,
+#plotselection label {
+    text-align: right;
+}
+
+.Tabs ul {
+    padding: 0px;
+    margin: 0px;
+    margin-left: 10px;
+    list-style-type: none;
+}
+
+.Tabs ul li {
+    display: inline-block;
+    float: left;
+    height: 24px;
+    cursor: pointer;
+    margin-bottom: 21px;
+}
+
+.Tabs ul li a {
+    position: relative;
+    margin-top: 16px;
+    display: block;
+    margin-left: 6px;
+    line-height: 28px;
+    padding-left: 10px;
+    padding-right: 10px;
+    background: #fff;
+    z-index: 9999;
+    border: 1px solid #ccc;
+    -moz-border-radius-topleft: 3px;
+    border-top-left-radius: 3px;
+    -moz-border-radius-topright: 3px;
+    border-top-right-radius: 3px;
+    color: #444;
+    text-decoration: none;
+    font-weight: bold;
+}
+
+.Tabs ul li.activeGpTab a {
+    background: #eee;
+    color: #000;
+    position: relative;
+    top: 1px;
+    border-bottom: 0px;
+}
+
+.Tabs ul li a:hover {
+    color: #222;
+    background: #ccc;
+}
+
+.Tabs #groupcontent,
+.Tabs .group_showing {
+    padding: 0 15px;
+    clear:both;
+    line-height:19px;
+    position: relative;
+    top: 20px;
+    z-index: 5;
+    height: 150px;
+    overflow: hidden;
+}
+
+.Tabs ul li a img {
+    position: relative;
+    top: 1px;
+    left: 4px;
+}
+
+.group_showing {
+    display: block;
+    height: 210px;
+    /*overflow-y: scroll;*/
+    padding: 5px 0 0 20px;
+    border: 1px solid #ccc;
+    -moz-border-radius: 3px;
+    -webkit-border-radius: 3px;
+    border-radius: 3px;   
+}
+
+.node_cbs {
+    overflow-y: scroll;
+    display: block;
+    height: 169px;
+}
+
+#outerplot {
+    min-height: 528px;
+    height: auto !important;
+    height: 528px;
+}
+
+#plot {
+    text-align: center;
+}
+
+#plot hr {
+    margin-bottom: 10px;
+}
+
+#lextend {
+    -moz-border-radius: 3px 0 0 3px;
+    -webkit-border-radius: 3px 0 0 3px;
+    border-radius: 3px 0 0 3px;
+}
+
+#rextend {
+    -moz-border-radius: 0 3px 3px 0;
+    -webkit-border-radius: 0 3px 3px 0;
+    border-radius: 0 3px 3px 0;
+}
+
+.group_showing {
+    margin-bottom: 20px;
+}
+
+
+.group_showing label {
+    color: #000;
+    display: inline;
+    font-family: "New Courier", Monospace;
+    cursor: pointer;
+    -moz-border-radius: 5px;
+    -webkit-border-radius: 5px;
+    border-radius: 5px;
+}
+
+.node_cbs label {
+    color: #ccc;
+    display: inline-block;
+    margin: 0 5px 5px 0;
+    padding: 4px;
+    font-family: "New Courier", Monospace;
+    cursor: pointer;
+    background: #eee;
+    -moz-border-radius: 5px;
+    -webkit-border-radius: 5px;
+    border-radius: 5px;
+}
+
+.group_showing label:hover {
+    background: #ddd;
+}
+
+.mycolumn {
+    float: left;
+    display: inline;
+}
+
+#lextend,
+#rextend,
+#larrow,
+#rarrow {
+    width: 27px;
+    height: 48px;
+    margin-bottom: 4px !important;
+}
+
+#timeTabsBasic label.button,
+#lextend,
+#rextend,
+#larrow,
+#rarrow {
+    /* remove color gradient from these buttons */
+    background: #eee;
+}
+
+#timeTabsBasic label.button,
+#larrow,
+#rarrow {
+    -moz-border-radius: 0;
+    -webkit-border-radius: 0;
+    border-radius: 0;
+}
+
+#timeTabsBasic label.button,
+#timeTabsBasic input[type="button"] {
+    margin: 0;
+    padding: 4px;
+    font-size: 80%;
+}
+
+#timeTabsBasic label.button {
+    text-align: center;
+    width: 47px;
+}
+
+#timeTabsBasic input[type="radio"],
+.node_cbs input[type="checkbox"] {
+   display: none;
+}
+
+#timeTabsBasic input[type="radio"]:checked + label.button {
+    border: 1px solid #666;
+    background: #ccc; /* Old browsers */
+}
+
+
+.node_cbs input[type="checkbox"]:checked + label {
+    color: #999;
+}
+
+.group_showing input[type="checkbox"]:checked + label {
+    color: #000;
+}
+
+.group_showing input[type="button"] {
+    margin: 0 0 2px 16px;
+    padding: 2px 8px;
+}
+
+.group_showing hr {  
+margin: 5px 0px 10px; 
+}
+
+.allnone {
+    margin: 2px 2px 5px 0;
+}
+
+.basicadv,
+.allnone {
+    font-size: 80% !important;
+    text-align: left;
+}
+
+.basicadv a,
+.allnone a {
+    text-decoration: none;
+    cursor: pointer;
+}
+
+#timedisplay {
+    text-align: right;
+    margin-top: 26px;
+}
+
+#userdisplay {
+    margin-top: 1px;
+    min-height: 13px;
+    height: auto !important;
+}
+
+#timedisplay p,
+#userdisplay p {
+    font-size: 80%;
+}
+
+#opts {
+    text-align: right;
+}
+
+#opts input[type="checkbox"] {
+    -webkit-appearance: checkbox;
+}
+
+#opts input[type="radio"],
+#radioplt input[type="radio"] {
+    -webkit-appearance: radio;
+}
+
+#opts input[type="checkbox"] + label,
+#opts input[type="radio"] + label,
+#radioplt input[type="radio"] + label {
+    display: inline;
+}
+
+#copyright {
+    text-align: center;
+}
+
+/* #Media Queries
+================================================== */
+
+	/* Smaller than standard 960 (devices and browsers) */
+	@media only screen and (max-width: 959px) {}
+
+	/* Tablet Portrait size to standard 960 (devices and browsers) */
+	@media only screen and (min-width: 768px) and (max-width: 959px) {}
+
+	/* All Mobile Sizes (devices and browser) */
+	@media only screen and (max-width: 767px) {}
+
+	/* Mobile Landscape Size to Tablet Portrait (devices and browsers) */
+	@media only screen and (min-width: 480px) and (max-width: 767px) {}
+
+	/* Mobile Portrait Size to Mobile Landscape Size (devices and browsers) */
+	@media only screen and (max-width: 479px) {}
+
+
+/* #Font-Face
+================================================== */
+/* 	This is the proper syntax for an @font-face file
+		Just create a "fonts" folder at the root,
+		copy your FontName into code below and remove
+		comment brackets */
+
+/*	@font-face {
+	    font-family: 'FontName';
+	    src: url('../fonts/FontName.eot');
+	    src: url('../fonts/FontName.eot?iefix') format('eot'),
+	         url('../fonts/FontName.woff') format('woff'),
+	         url('../fonts/FontName.ttf') format('truetype'),
+	         url('../fonts/FontName.svg#webfontZam02nTh') format('svg');
+	    font-weight: normal;
+	    font-style: normal; }
+*/
\ No newline at end of file
diff --git a/www/hedgehog/www/stylesheets/skeleton.css b/www/hedgehog/www/stylesheets/skeleton.css
new file mode 100644
index 0000000..5db3d38
--- /dev/null
+++ b/www/hedgehog/www/stylesheets/skeleton.css
@@ -0,0 +1,242 @@
+/*
+* Skeleton V1.2
+* Copyright 2011, Dave Gamache
+* www.getskeleton.com
+* Free to use under the MIT license.
+* http://www.opensource.org/licenses/mit-license.php
+* 6/20/2012
+*/
+
+
+/* Table of Contents
+==================================================
+    #Base 960 Grid
+    #Tablet (Portrait)
+    #Mobile (Portrait)
+    #Mobile (Landscape)
+    #Clearing */
+
+
+
+/* #Base 960 Grid
+================================================== */
+
+    .container                                  { position: relative; width: 960px; margin: 0 auto; padding: 0; }
+    .container .column,
+    .container .columns                         { float: left; display: inline; margin-left: 10px; margin-right: 10px; }
+    .row                                        { margin-bottom: 20px; }
+
+    /* Nested Column Classes */
+    .column.alpha, .columns.alpha               { margin-left: 0; }
+    .column.omega, .columns.omega               { margin-right: 0; }
+
+    /* Base Grid */
+    .container .one.column,
+    .container .one.columns                     { width: 40px;  }
+    .container .two.columns                     { width: 100px; }
+    .container .three.columns                   { width: 160px; }
+    .container .four.columns                    { width: 220px; }
+    .container .five.columns                    { width: 280px; }
+    .container .six.columns                     { width: 340px; }
+    .container .seven.columns                   { width: 400px; }
+    .container .eight.columns                   { width: 460px; }
+    .container .nine.columns                    { width: 520px; }
+    .container .ten.columns                     { width: 580px; }
+    .container .eleven.columns                  { width: 640px; }
+    .container .twelve.columns                  { width: 700px; }
+    .container .thirteen.columns                { width: 760px; }
+    .container .fourteen.columns                { width: 820px; }
+    .container .fifteen.columns                 { width: 880px; }
+    .container .sixteen.columns                 { width: 940px; }
+
+    .container .one-third.column                { width: 300px; }
+    .container .two-thirds.column               { width: 620px; }
+
+    /* Offsets */
+    .container .offset-by-one                   { padding-left: 60px;  }
+    .container .offset-by-two                   { padding-left: 120px; }
+    .container .offset-by-three                 { padding-left: 180px; }
+    .container .offset-by-four                  { padding-left: 240px; }
+    .container .offset-by-five                  { padding-left: 300px; }
+    .container .offset-by-six                   { padding-left: 360px; }
+    .container .offset-by-seven                 { padding-left: 420px; }
+    .container .offset-by-eight                 { padding-left: 480px; }
+    .container .offset-by-nine                  { padding-left: 540px; }
+    .container .offset-by-ten                   { padding-left: 600px; }
+    .container .offset-by-eleven                { padding-left: 660px; }
+    .container .offset-by-twelve                { padding-left: 720px; }
+    .container .offset-by-thirteen              { padding-left: 780px; }
+    .container .offset-by-fourteen              { padding-left: 840px; }
+    .container .offset-by-fifteen               { padding-left: 900px; }
+
+
+
+/* #Tablet (Portrait)
+================================================== */
+
+    /* Note: Design for a width of 768px */
+
+    @media only screen and (min-width: 768px) and (max-width: 959px) {
+        .container                                  { width: 768px; }
+        .container .column,
+        .container .columns                         { margin-left: 10px; margin-right: 10px;  }
+        .column.alpha, .columns.alpha               { margin-left: 0; margin-right: 10px; }
+        .column.omega, .columns.omega               { margin-right: 0; margin-left: 10px; }
+        .alpha.omega                                { margin-left: 0; margin-right: 0; }
+
+        .container .one.column,
+        .container .one.columns                     { width: 28px; }
+        .container .two.columns                     { width: 76px; }
+        .container .three.columns                   { width: 124px; }
+        .container .four.columns                    { width: 172px; }
+        .container .five.columns                    { width: 220px; }
+        .container .six.columns                     { width: 268px; }
+        .container .seven.columns                   { width: 316px; }
+        .container .eight.columns                   { width: 364px; }
+        .container .nine.columns                    { width: 412px; }
+        .container .ten.columns                     { width: 460px; }
+        .container .eleven.columns                  { width: 508px; }
+        .container .twelve.columns                  { width: 556px; }
+        .container .thirteen.columns                { width: 604px; }
+        .container .fourteen.columns                { width: 652px; }
+        .container .fifteen.columns                 { width: 700px; }
+        .container .sixteen.columns                 { width: 748px; }
+
+        .container .one-third.column                { width: 236px; }
+        .container .two-thirds.column               { width: 492px; }
+
+        /* Offsets */
+        .container .offset-by-one                   { padding-left: 48px; }
+        .container .offset-by-two                   { padding-left: 96px; }
+        .container .offset-by-three                 { padding-left: 144px; }
+        .container .offset-by-four                  { padding-left: 192px; }
+        .container .offset-by-five                  { padding-left: 240px; }
+        .container .offset-by-six                   { padding-left: 288px; }
+        .container .offset-by-seven                 { padding-left: 336px; }
+        .container .offset-by-eight                 { padding-left: 384px; }
+        .container .offset-by-nine                  { padding-left: 432px; }
+        .container .offset-by-ten                   { padding-left: 480px; }
+        .container .offset-by-eleven                { padding-left: 528px; }
+        .container .offset-by-twelve                { padding-left: 576px; }
+        .container .offset-by-thirteen              { padding-left: 624px; }
+        .container .offset-by-fourteen              { padding-left: 672px; }
+        .container .offset-by-fifteen               { padding-left: 720px; }
+    }
+
+
+/*  #Mobile (Portrait)
+================================================== */
+
+    /* Note: Design for a width of 320px */
+
+    @media only screen and (max-width: 767px) {
+        .container { width: 300px; }
+        .container .columns,
+        .container .column { margin: 0; }
+
+        .container .one.column,
+        .container .one.columns,
+        .container .two.columns,
+        .container .three.columns,
+        .container .four.columns,
+        .container .five.columns,
+        .container .six.columns,
+        .container .seven.columns,
+        .container .eight.columns,
+        .container .nine.columns,
+        .container .ten.columns,
+        .container .eleven.columns,
+        .container .twelve.columns,
+        .container .thirteen.columns,
+        .container .fourteen.columns,
+        .container .fifteen.columns,
+        .container .sixteen.columns,
+        .container .one-third.column,
+        .container .two-thirds.column  { width: 300px; }
+
+        /* Offsets */
+        .container .offset-by-one,
+        .container .offset-by-two,
+        .container .offset-by-three,
+        .container .offset-by-four,
+        .container .offset-by-five,
+        .container .offset-by-six,
+        .container .offset-by-seven,
+        .container .offset-by-eight,
+        .container .offset-by-nine,
+        .container .offset-by-ten,
+        .container .offset-by-eleven,
+        .container .offset-by-twelve,
+        .container .offset-by-thirteen,
+        .container .offset-by-fourteen,
+        .container .offset-by-fifteen { padding-left: 0; }
+
+    }
+
+
+/* #Mobile (Landscape)
+================================================== */
+
+    /* Note: Design for a width of 480px */
+
+    @media only screen and (min-width: 480px) and (max-width: 767px) {
+        .container { width: 420px; }
+        .container .columns,
+        .container .column { margin: 0; }
+
+        .container .one.column,
+        .container .one.columns,
+        .container .two.columns,
+        .container .three.columns,
+        .container .four.columns,
+        .container .five.columns,
+        .container .six.columns,
+        .container .seven.columns,
+        .container .eight.columns,
+        .container .nine.columns,
+        .container .ten.columns,
+        .container .eleven.columns,
+        .container .twelve.columns,
+        .container .thirteen.columns,
+        .container .fourteen.columns,
+        .container .fifteen.columns,
+        .container .sixteen.columns,
+        .container .one-third.column,
+        .container .two-thirds.column { width: 420px; }
+    }
+
+
+/* #Clearing
+================================================== */
+
+    /* Self Clearing Goodness */
+    .container:after { content: "\0020"; display: block; height: 0; clear: both; visibility: hidden; }
+
+    /* Use clearfix class on parent to clear nested columns,
+    or wrap each row of columns in a <div class="row"> */
+    .clearfix:before,
+    .clearfix:after,
+    .row:before,
+    .row:after {
+      content: '\0020';
+      display: block;
+      overflow: hidden;
+      visibility: hidden;
+      width: 0;
+      height: 0; }
+    .row:after,
+    .clearfix:after {
+      clear: both; }
+    .row,
+    .clearfix {
+      zoom: 1; }
+
+    /* You can also use a <br class="clear" /> to clear columns */
+    .clear {
+      clear: both;
+      display: block;
+      overflow: hidden;
+      visibility: hidden;
+      width: 0;
+      height: 0;
+    }

Fixed code: #Vagrant
.vagrant
._*

# Generated automatically
/build
Makefile.in
/*.cache
/aclocal.m4
/compile
/config.guess
/config.h.*
/config.sub
/configure
/depcomp
/install-sh
/ltmain.sh
/missing
/m4/libtool.m4
/m4/lt*
/src/rssac_server/*.pyc
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#

Hedgehog 2.4.2 - 2019-01-08
----------------------------
* Improve validation of URL parameters to fix a SQL injection attack. The attack
  can potentially leak data available to the user that the web server connects
  to the database as. In the default configuration this is the 'www-data'
  read-only user with access to only the Hedgehog database.


Hedgehog 2.4.1 - 2018-05-08
----------------------------
* The only change is an update to the packaging that switches the postgres 9.3
  packages from 'depends' to 'suggests' to allow the user to choose the DB 
  version.


Hedgehog 2.4.0 - 2017-08-21
----------------------------
* Improved error handling when trying to insert invalid data into db 
* Renamed Node Statistics to Query Statistics in plot dropdown
* New configuration option to hide 'node' selection on web interface
* Add new graph for Server IP address vs transport (UDP/TCP)
* Changed license to MPL
* Added ability to re-name and remove nodes
* Tested with Postgesql-9.6


Hedgehog 2.3.0 - 2016-09-12
----------------------------
* Add ability to filter the server addresses displayed in the 'By server IP
  address' plot. Controlled by the database_update_service_addr script.
* Add new 'By region' graph which is now the default. This graph always groups
  the results by Region. The 'By node' graph now always displays a line per node
  regardless of the node selection (previously it grouped by Region when all
  nodes were selected).
* Default plot is now a configuration option. To specify a plot use the 'Hedgehog
  internal plot name' as listed in the User Guide.
* Update RSSAC report header format to match RSSC002v3 report.
* Enhancements to datafiles_create_summary script.


Hedgehog 2.2.0 - 2016-08-31
-----------------------------
* Added new dataset and plot dnskey_vs_asn


Hedgehog 2.1.1 - 2016-08-16
-----------------------------
* Fix issue with database_manage_partitions script where generating tables for
  -09 and -10 months didn't work
* Add additional check on input parameters
* Modify instructions due to issue with latest build of rcpp
* Migrate to install from packages only from 2.1.1


Hedgehog 2.1.0 - 2016-06-20
------------------------------
* Production release of 'Daurian'


Hedgehog 2.1.0rc3 - 2016-06-01
------------------------------
* Add option to specify number of processes for refile_and_grok to use
* Fix connection issue in update_tlds_from_zone script
* Make hedgehogctl produce correct tools list for split installs
* Minor documentation fixes


Hedgehog 2.1.0rc2 - 2016-05-19
------------------------------
* Add default server for drop down as a config option
* Fix alphabetical sorting of Country/City/Instance in node tab
* Fix detection of correct user when running from cron jobs


Hedgehog 2.1.0rc1 - 2016-05-09
------------------------------
* Updated packaging now that RPostgreSQL is available (thanks to Ondrej Sury)


Hedgehog 2.1.0b2 - 2016-04-20
-----------------------------

* Add release name of "Daurian" for 2.1
* Add script to update TLDs from IANA
* Updates to documentation
* Bug fix to handling of nodes.csv input
* Ensure output format consistent with latest RSSAC02 specs
* Fix issues with rssacd
* Support '.' in server names
* Move webdav config
* Fix issues with jquery dependancies


Hedgehog 2.1.0b1 - 2016-02-01 
------------------------------

Features:
* Added new graphs for different types of TLD (QTYPE and RCODE for most
  popular TLD). Some support for displaying IDN in their unicode glyphs.
* Added new graph to display Geo-locations of querying subnets and nodes
* Added new graphs to display query sources as BGP prefixes or ASN
* Added new graphs to group nodes by Country/City/Instance
* Added new graph to display Destination Address vs Query rate
* Added option to group nodes in selection tab by Country/City/Instance and 
  config option for default choice (specified in nodes.csv).
* Added rssacd demon to collect rssac load-time and zone-size data. Requires
  management IP Address of node to be specified in nodes.csv
* Added GeoLite2 data created by MaxMind, available from
  http://www.maxmind.com
* Added static link URI to provide access to raw plot files
* Updated database_update_nodes script to support new fields. NOTE: New format
  is not backwards compatible!
* Multiple remote Web front ends now supported
* New config option to customise 'Support' link on Web page

Maintenance/Bug fixes:
* Improve DB performance by adding further indexes
* Change to rssac scripts. Processing of unique_sources data now done by
  'database_process_rssac_data' script which must be explicitly called.
* Wrote check_user_perms to do all user checks, check db access and data dir.
  access. Used in new database_update_nodes will gradually be used by all tools.
* Improved chart sizes, font sizes and data ordering in some graphs
* Significant enhancements to installation and management scripts and 
  configuration files.
* Fixed appearance of gvis column chart vAxis
* Fixed error in rssac reports where ".root-servers.org" was used instead of 
 ".root-servers.net". Also TCP length field is no longer included in the size
  of the zone. 


Hedgehog 2.0.0rc2 - 2015-07-09
------------------------------

* Add documentation source for packaging.

Hedgehog 2.0.0rc1 - 2015-05-29
------------------------------

* Allow building of just the importer or just the web interface by adding
  --disable-web and --disable-data-manager options to configure
* Installed tools in libexec/hedgehog/ and removed hedgehog prefix and 
  .sh extension
* Created a hedgehogctl control script to wrap the tools.
* Renamed several tools and made arguments consistent. Separate the RSSAC report
  scripts to better align with split installs
* Improved install instructions
* Removed the install-rpg target and made installation of the RPostgreSQLHelper
  consistent with other R packages that need installing.
* Add missing check for SED in configure

Hedgehog 2.0.0b2 - 2015-05-05
-----------------------------

* Support for Vagrant Provisioning (thanks melalj)
* Various installation changes to support packaging (thanks arhbkb)
* Changed the yaml parsing to use the Perl YAML::Tiny library
* Changed the default_interactive_plot_type to svg (instead of flash)
* Static Traffic Volume plot is now faceted by Transport vs IP version
* Traffic Query/Response Difference plots added (requires R package dplyr)
* RPostgreSQLHelper installed as separate package. This requires running
 'make install ; make install-rpg' on installation now.
* Added vertical axis labels to googlevis line charts
* Added node_id index to data tables
* Added support for connecting to a remote database and password support
* Fixed color palette in annotated charts
* Work around for bug: mages/googleVis#14
* Improved old xml tar script
* Improvements to database creation and upgrade scripts

Hedgehog 2.0.0b1 - 2014-09-09
-----------------------------

* Database schema changes: Plot table split into 3 and replaced by a view.
  Added explicit handling of plots based on 2 datasets.
  ****** Database version bumped - not backwards compatible *****
* Added new config option 'default_interactive_plot_type' to 
  hedgehog_gui.yaml. This can be used to generate SVG interactive plots
  instead of the default interactive plots which require Flash.
* Issue-#1: hedgehog_gui.yaml configuration file is now installed in
  <prefix>/etc/hedgeghog
* Database version mismatch now reported correctly in the GUI


Hedgehog 2.0.0a1 - 2014-08-27
----------------------------

* Initial open source alpha release including:
  * Data importer (refile_and_grok.sh)
    - DAT-> Postgres database importer
    - XML-> Postgres database importer
    - XML-> DAT converter
  * Database schema and scripts
  * Web GUI
  * Support for multiple servers
  * Support for import of RSSAC data (from XML only)
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#

KNOWN ISSUES/LIMITATIONS:
-------------------------

* The full list of possible plots are displayed via the drop down menu in the
  GUI, regardless of whether there is data in the database for the plot

* A configuration option is available (default_interactive_plot_type in 
  hedgehog_gui.yaml) which can specify the system to produce SVG or 
  Flash plots for the interactive plots (google charts). There is a known 
  issue with the SVG plots where long legends are not wrapped properly:
  https://code.google.com/p/google-visualization-api-issues/issues/detail?id=1536
  This affects, for example, the 'by_node' plot. However, we prefer SVG
  to flash given all the flash vulnerabilities of late. 

* Support for RSSAC reports is added for XML imports only, however extended 
  RCODES cannot be collected by the existing DSC collector so only base RCODES
  are collected and displayed. If the DSC collector is extended then further
  work must be done in Hedgehog to also support extended RCODES

* The Unique Sources plot is not available until after the RSSAC report has
  been generated for the date in question.

* If a user touches any controls while a query is waiting to return
  then the 'generate graph' button becomes enabled again and a second
  plot can be generated that will then over write the first.

* No static plots are produced for the GEO type plots.The geochart is ignoring 
  the instruction in the html not to show the color code in the tooltips.

* Server, node and group and instance names may only contain alphanumeric 
  characters, full stops (.) and hyphens (-). Country and city names can
  only contain alphanumeric characters, spaces and hyphens (-)
  
* The Busiest client locations by city plot does't support passing the required
  API key to the script element - see
  https://developers.google.com/maps/documentation/javascript/error-messages#missing-key-map-error
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

This project contains code from the following projects:

Boost
# Boost Software License - Version 1.0 - August 17th, 2003
# 
# Permission is hereby granted, free of charge, to any person or organization
# obtaining a copy of the software and accompanying documentation covered by
# this license (the "Software") to use, reproduce, display, distribute,
# execute, and transmit the Software, and to prepare derivative works of the
# Software, and to permit third-parties to whom the Software is furnished to
# do so, all subject to the following:
# 
# The copyright notices in the Software and this entire statement, including
# the above license grant, this restriction and the following disclaimer,
# must be included in all copies of the Software, in whole or in part, and
# all derivative works of the Software, unless such copies or derivative
# works are solely in the form of machine-executable object code generated by
# a source language processor.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
# SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
# FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.


boost.m4
# boost.m4: Locate Boost headers and libraries for autoconf-based projects.
# Copyright (C) 2007, 2008, 2009, 2010, 2011  Benoit Sigoure <tsuna@lrde.epita.fr>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Additional permission under section 7 of the GNU General Public
# License, version 3 ("GPLv3"):
#
# If you convey this file as part of a work that contains a
# configuration script generated by Autoconf, you may do so under
# terms of your choice.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

Skeleton CSS
/*
* Skeleton V1.2
* Copyright 2011, Dave Gamache
* www.getskeleton.com
* Free to use under the MIT license.
* http://www.opensource.org/licenses/mit-license.php
* 6/20/2012
*/

############################################################################

Mozilla Public License Version 2.0
==================================

1. Definitions
--------------

1.1. "Contributor"
    means each individual or legal entity that creates, contributes to
    the creation of, or owns Covered Software.

1.2. "Contributor Version"
    means the combination of the Contributions of others (if any) used
    by a Contributor and that particular Contributor's Contribution.

1.3. "Contribution"
    means Covered Software of a particular Contributor.

1.4. "Covered Software"
    means Source Code Form to which the initial Contributor has attached
    the notice in Exhibit A, the Executable Form of such Source Code
    Form, and Modifications of such Source Code Form, in each case
    including portions thereof.

1.5. "Incompatible With Secondary Licenses"
    means

    (a) that the initial Contributor has attached the notice described
        in Exhibit B to the Covered Software; or

    (b) that the Covered Software was made available under the terms of
        version 1.1 or earlier of the License, but not also under the
        terms of a Secondary License.

1.6. "Executable Form"
    means any form of the work other than Source Code Form.

1.7. "Larger Work"
    means a work that combines Covered Software with other material, in 
    a separate file or files, that is not Covered Software.

1.8. "License"
    means this document.

1.9. "Licensable"
    means having the right to grant, to the maximum extent possible,
    whether at the time of the initial grant or subsequently, any and
    all of the rights conveyed by this License.

1.10. "Modifications"
    means any of the following:

    (a) any file in Source Code Form that results from an addition to,
        deletion from, or modification of the contents of Covered
        Software; or

    (b) any new file in Source Code Form that contains any Covered
        Software.

1.11. "Patent Claims" of a Contributor
    means any patent claim(s), including without limitation, method,
    process, and apparatus claims, in any patent Licensable by such
    Contributor that would be infringed, but for the grant of the
    License, by the making, using, selling, offering for sale, having
    made, import, or transfer of either its Contributions or its
    Contributor Version.

1.12. "Secondary License"
    means either the GNU General Public License, Version 2.0, the GNU
    Lesser General Public License, Version 2.1, the GNU Affero General
    Public License, Version 3.0, or any later versions of those
    licenses.

1.13. "Source Code Form"
    means the form of the work preferred for making modifications.

1.14. "You" (or "Your")
    means an individual or a legal entity exercising rights under this
    License. For legal entities, "You" includes any entity that
    controls, is controlled by, or is under common control with You. For
    purposes of this definition, "control" means (a) the power, direct
    or indirect, to cause the direction or management of such entity,
    whether by contract or otherwise, or (b) ownership of more than
    fifty percent (50%) of the outstanding shares or beneficial
    ownership of such entity.

2. License Grants and Conditions
--------------------------------

2.1. Grants

Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:

(a) under intellectual property rights (other than patent or trademark)
    Licensable by such Contributor to use, reproduce, make available,
    modify, display, perform, distribute, and otherwise exploit its
    Contributions, either on an unmodified basis, with Modifications, or
    as part of a Larger Work; and

(b) under Patent Claims of such Contributor to make, use, sell, offer
    for sale, have made, import, and otherwise transfer either its
    Contributions or its Contributor Version.

2.2. Effective Date

The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.

2.3. Limitations on Grant Scope

The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:

(a) for any code that a Contributor has removed from Covered Software;
    or

(b) for infringements caused by: (i) Your and any other third party's
    modifications of Covered Software, or (ii) the combination of its
    Contributions with other software (except as part of its Contributor
    Version); or

(c) under Patent Claims infringed by Covered Software in the absence of
    its Contributions.

This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).

2.4. Subsequent Licenses

No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).

2.5. Representation

Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.

2.6. Fair Use

This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.

2.7. Conditions

Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.

3. Responsibilities
-------------------

3.1. Distribution of Source Form

All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.

3.2. Distribution of Executable Form

If You distribute Covered Software in Executable Form then:

(a) such Covered Software must also be made available in Source Code
    Form, as described in Section 3.1, and You must inform recipients of
    the Executable Form how they can obtain a copy of such Source Code
    Form by reasonable means in a timely manner, at a charge no more
    than the cost of distribution to the recipient; and

(b) You may distribute such Executable Form under the terms of this
    License, or sublicense it under different terms, provided that the
    license for the Executable Form does not attempt to limit or alter
    the recipients' rights in the Source Code Form under this License.

3.3. Distribution of a Larger Work

You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).

3.4. Notices

You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.

3.5. Application of Additional Terms

You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.

4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------

If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.

5. Termination
--------------

5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.

5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.

5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.

************************************************************************
*                                                                      *
*  6. Disclaimer of Warranty                                           *
*  -------------------------                                           *
*                                                                      *
*  Covered Software is provided under this License on an "as is"       *
*  basis, without warranty of any kind, either expressed, implied, or  *
*  statutory, including, without limitation, warranties that the       *
*  Covered Software is free of defects, merchantable, fit for a        *
*  particular purpose or non-infringing. The entire risk as to the     *
*  quality and performance of the Covered Software is with You.        *
*  Should any Covered Software prove defective in any respect, You     *
*  (not any Contributor) assume the cost of any necessary servicing,   *
*  repair, or correction. This disclaimer of warranty constitutes an   *
*  essential part of this License. No use of any Covered Software is   *
*  authorized under this License except under this disclaimer.         *
*                                                                      *
************************************************************************

************************************************************************
*                                                                      *
*  7. Limitation of Liability                                          *
*  --------------------------                                          *
*                                                                      *
*  Under no circumstances and under no legal theory, whether tort      *
*  (including negligence), contract, or otherwise, shall any           *
*  Contributor, or anyone who distributes Covered Software as          *
*  permitted above, be liable to You for any direct, indirect,         *
*  special, incidental, or consequential damages of any character      *
*  including, without limitation, damages for lost profits, loss of    *
*  goodwill, work stoppage, computer failure or malfunction, or any    *
*  and all other commercial damages or losses, even if such party      *
*  shall have been informed of the possibility of such damages. This   *
*  limitation of liability shall not apply to liability for death or   *
*  personal injury resulting from such party's negligence to the       *
*  extent applicable law prohibits such limitation. Some               *
*  jurisdictions do not allow the exclusion or limitation of           *
*  incidental or consequential damages, so this exclusion and          *
*  limitation may not apply to You.                                    *
*                                                                      *
************************************************************************

8. Litigation
-------------

Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.

9. Miscellaneous
----------------

This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.

10. Versions of the License
---------------------------

10.1. New Versions

Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.

10.2. Effect of New Versions

You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.

10.3. Modified Versions

If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).

10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses

If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.

Exhibit A - Source Code Form License Notice
-------------------------------------------

  This Source Code Form is subject to the terms of the Mozilla Public
  License, v. 2.0. If a copy of the MPL was not distributed with this
  file, You can obtain one at http://mozilla.org/MPL/2.0/.

If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.

You may add additional accurate notices of copyright ownership.

Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------

  This Source Code Form is "Incompatible With Secondary Licenses", as
  defined by the Mozilla Public License, v. 2.0.
ACLOCAL_AMFLAGS = -I m4

if DM
D = src
else
D =
endif

if WEB
W = www
else
W =
endif

SUBDIRS = conf tools $(W) $(D)

MAINTAINERCLEANFILES = $(srcdir)/Makefile.in

hh_web_conf_dir    = hedgehog/conf
hh_web_source_dir  = hedgehog/www
hh_data_dir        = hedgehog/data
hh_plotcache_dir   = hedgehog/www/plotcache
hh_rssac_dir       = hedgehog/www/rssac
hh_conf_dir        = hedgehog
hh_conf_file       = hedgehog.yaml

# Used to substitute into the hedgehog.yaml conf file and the hedgehog.conf file
subst_HH_DIRS = sed -e "s|@HH_WEB_CONF@|@datadir@/$(hh_web_conf_dir)|" \
					-e "s|@HH_WEB_SOURCE@|@datadir@/$(hh_web_source_dir)|" \
					-e "s|@HH_DATA_DIR@|@localstatedir@/$(hh_data_dir)|" \
					-e "s|@HH_PLOTCACHE@|@localstatedir@/$(hh_plotcache_dir)|" \
					-e "s|@HH_CONF@|@sysconfdir@/$(hh_conf_dir)|" \
					-e "s|@HH_RSSAC@|@localstatedir@/$(hh_rssac_dir)|"

#Used to substitute into startupR.in and hedgeghog_conf_read.sh
subst_HH_CONF_FILE = sed -e "s|@HH_CONF_FILE@|@sysconfdir@/$(hh_conf_dir)/$(hh_conf_file)|" 

export subst_HH_DIRS
export subst_HH_CONF_FILE
export hh_web_conf_dir
Hedgehog

DESCRIPTION
-----------
*Please note that Hedgehog is no longer actively developed.* 
The DNS-STATS project is now focussing on
a newer data collection (dns-stats-compactor) and presentation solution - see the
http://www.dns-stats.org website.

Hedgehog is a visualisation tool for DNS statistics that consumes data
acquired with the DSC collector.

Hedgehog was initially developed for ICANN by Sinodun IT and is now released
open source. This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, you can obtain one at https://mozilla.org/MPL/2.0/.


REGISTRATION
-------------
If you use Hedgehog, please consider subscribing to the Users Mailing list:
https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
and providing feedback. It helps us to know how Hedgehog is being used!

OVERVIEW
--------
For general information and the User guide see the Hedgehog website:
https://github.com/dns-stats/hedgehog/wiki

USER GUIDE
----------
Read the User_Guide.pdf in the docs directory for the lastest version or look
online: https://github.com/dns-stats/hedgehog/wiki/Documentation

PACKAGES
---------
The latest Hedgehog packages are available from ppa:dns-stats/presenter

SOURCE INSTALLATION
------------
This is no longer officially supported.

VAGRANT INSTALLATION
------------
You can load Hedgehog locally using a virtual machine and Vagrant 
(https://www.vagrantup.com/downloads) â Requires also VirtualBox.
Execute `vagrant up` to build the vm and run provisionning scripts 
(dependencies installation, compilation, configuration).
Once completed, you can access to the server using this url: 
`http://localhost:8080/hedgehog`

SUPPORT
-------
- Mailing list: https://mm.dns-stats.org/mailman/listinfo/dns-stats-users
- Issue tracker: https://github.com/dns-stats/hedgehog/issues
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.box = "ubuntu/trusty64"
  config.vm.box_check_update = false

  config.vm.network :forwarded_port, guest: 80, host: 8080, host_ip: "127.0.0.1"
  config.vm.network :private_network, ip: '192.168.50.50'

  config.vm.provider "virtualbox" do |v|
    v.memory = 2048
    v.cpus = 2
  end

  config.vm.synced_folder '.', '/vagrant'

  config.vm.provision "shell", path: "provision/dependencies.sh"
  config.vm.provision "shell", path: "provision/compile.sh"
  config.vm.provision "shell", path: "provision/launch.sh"

end
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#

autoreconf --install --force
CLEANFILES = $(localstate_DATA) $(noinst_DATA)

sysconfdir = @sysconfdir@/hedgehog

nobase_sysconf_DATA = $(srcdir)/tlds.d/cctlds.csv \
                      $(srcdir)/tlds.d/idn_cctlds.csv \
                      $(srcdir)/tlds.d/legacy_gtlds.csv \
                      $(srcdir)/tlds.d/new_gtlds-20131023-20150611.csv \
                      $(srcdir)/tlds.d/new_gtlds-20150614-20151223.csv

noinst_DATA = hedgehog.yaml

hedgehog.yaml: $(srcdir)/hedgehog.yaml.in Makefile
	$(subst_HH_DIRS) < $(srcdir)/hedgehog.yaml.in > hedgehog.yaml

EXTRA_DIST = (srcdir)/hedgehog.yaml.in

install-data-hook:
	test -d ${DESTDIR}${sysconfdir} || mkdir -p ${DESTDIR}${sysconfdir}
	test -f ${DESTDIR}${sysconfdir}/hedgehog.yaml || \
		${INSTALL_DATA} -m 644 hedgehog.yaml ${DESTDIR}${sysconfdir}
	${INSTALL_DATA} -m 644 hedgehog.yaml ${DESTDIR}${sysconfdir}/hedgehog.yaml.sample
	test -f ${DESTDIR}${sysconfdir}/nodes.csv || \
		${INSTALL_DATA} -m 644 $(srcdir)/nodes.csv ${DESTDIR}${sysconfdir}
	test -f ${DESTDIR}${sysconfdir}/service_addr.csv || \
		${INSTALL_DATA} -m 644 $(srcdir)/service_addr.csv ${DESTDIR}${sysconfdir}
	${INSTALL_DATA} -m 644 $(srcdir)/nodes.csv ${DESTDIR}${sysconfdir}/nodes.csv.sample
## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
##
## This Source Code Form is subject to the terms of the Mozilla Public
## License, v. 2.0. If a copy of the MPL was not distributed with this
## file, you can obtain one at https://mozilla.org/MPL/2.0/.

## Developed by Sinodun IT (www.sinodun.com)

## YAML config for Hedgehog
--- 
# Database parameters to be specified by the user - these must not be defaulted
database:
  host         : /var/run/postgresql # specify a host for the postgresql DB. If this begins with a slash,
                                     # it specifies the directory in which the socket file is stored.
  port         : 5432      # specify port for the postgresql DB.
  name         : hedgehog  # specify dbname for the postgresql DB.
  owner        : hedgehog  # specify a user to own the postgresql DB.
                           # [Required for Data Manager component]
  owner_pass   :           # specify a password for the owner user if needed.
  read_user    : www-data  # specify a read user for the postgresql DB.
                           # [Required for Web front-end component]
  read_pass    :           # specify a password for the read user if needed.
    
# Directory structure
# !! This information is currently auto-populated at compile time and it is _not_ recommended that users alter this !!
directories:
  conf         : @HH_CONF@          # directory of hedgehog configuration files
  data         : @HH_DATA_DIR@      # directory holding the XML for all the servers
  plotcache    : @HH_PLOTCACHE@     # location of the plotcache used by the web front end to hold image files
  rssac        : @HH_RSSAC@         # location of the rssac directory where the daily reports are stored
  web_conf     : @HH_WEB_CONF@      # configuration files for web front end
  web_source   : @HH_WEB_SOURCE@    # source files for web front end
...
######################### NODES CONFIGURATION FILE ###################
# Format is:     server-name, node-name, [node-group],[node-country],
#                [node-city],[node-instance],[node-ip]
# Server name, node name, group and instance fields can contain alphanumeric
# characters, full stops (.) and hyphens (-) only.
# Country and city fields can contain alphanumeric characters, spaces and
# hyphens (-) only.
# Nodes may be grouped and plotted by Country,City or Instance in 2.1.
# '#' is the comment character
# 
# server-name   - This must be unique and it must match the name used in the
#                 data directory path for uploaded files.
# node-name     - This must be unique for a given server and it must 
#                 match the name used in the data directory path for 
#                 uploaded files.
# node-group    - This is used in the node grouping tabs in the Hedgehog
#                 front end. It should be unique for a given server. 
#                 It is optional and if not
#                 specified the node is added to a group called 'Other'.
# node-country  - This is the country that the node is located in
#                 It is optional and if not specified 'No Country' is used.
# node-city     - This is the city that the node is located in
#                 It is optional and if not specified 'No City' is used.
#                 This is used to plot node locations in GEO plots
# node-instance - This is used to collect nodes into subgroups called 'Instances' 
#                 (e.g. nodes within the same city) for convenient selection
#                 and plotting. It should be unique for a given group. It is 
#                 optional and if not specified 'No Instance' is used.
# node-ip       - The management IP address of the node (IPv4 or IPv6)
#                 It is optional and if not specified is left blank. This is 
#                 required if RSSAC load-time/zone-size data is to be collected.
# rename-or-remove - This field is used to either rename or remove a node. It 
#                    is optional. It is only honored if the -r flag is set and
#                    must be empty if the -r flag is not used. 
#                    If this field contains the name of an existing node and the
#                    node-name contains a different name then the node is
#                    renamed. 
#                    If this field contains "remove" then the node and all 
#                    associated data is removed from the database. Data on disk
#                    is not touched.
#
# Example input: (uncomment the following)
#
#Server-A,Node-1,Region-1,Country-1,City-1,Instance-1,10.0.0.2,Node-a
#Server-A,Node-2,Region-1,Country-1,City-2,Instance-2,
#Server-A,Node-3,Region-1,Country-2,City-3,Instance-3,
#Server-A,Node-4,Region-1,Country-2,City-3,Instance-3,
#Server-A,Node-5,Region-2,Country-3,City-4,Instance-4,
#Server-A,Node-6,Region-2,Country-3,City-4,Instance-4,, remove
#Server-A,Node-7,Region-3,Country-3,City-4,Instance-5,10.0.0.3
#Server-B,Node-1,Region-1,Country-4,,,
#Server-B,Node-2,Region-2,,,Instance-6,
#Server-C,Node-5
######################################################################
######################### SERVICE ADDRESS CONFIGURATION FILE ###################
# Format is:     server-name, address (IPv4 or IPv6)
#
# L-Root,2001:500:9f::42
# L-Root,199.7.83.42
# L-Root,2001:500:3::42
# Format is: alabel, type

ac, ccTLD
ad, ccTLD
ae, ccTLD
af, ccTLD
ag, ccTLD
ai, ccTLD
al, ccTLD
am, ccTLD
an, ccTLD
ao, ccTLD
aq, ccTLD
ar, ccTLD
as, ccTLD
at, ccTLD
au, ccTLD
aw, ccTLD
ax, ccTLD
az, ccTLD
ba, ccTLD
bb, ccTLD
bd, ccTLD
be, ccTLD
bf, ccTLD
bg, ccTLD
bh, ccTLD
bi, ccTLD
bj, ccTLD
bm, ccTLD
bn, ccTLD
bo, ccTLD
br, ccTLD
bs, ccTLD
bt, ccTLD
bv, ccTLD
bw, ccTLD
by, ccTLD
bz, ccTLD
ca, ccTLD
cc, ccTLD
cd, ccTLD
cf, ccTLD
cg, ccTLD
ch, ccTLD
ci, ccTLD
ck, ccTLD
cl, ccTLD
cm, ccTLD
cn, ccTLD
co, ccTLD
cr, ccTLD
cu, ccTLD
cv, ccTLD
cw, ccTLD
cx, ccTLD
cy, ccTLD
cz, ccTLD
de, ccTLD
dj, ccTLD
dk, ccTLD
dm, ccTLD
do, ccTLD
dz, ccTLD
ec, ccTLD
ee, ccTLD
eg, ccTLD
er, ccTLD
es, ccTLD
et, ccTLD
eu, ccTLD
fi, ccTLD
fj, ccTLD
fk, ccTLD
fm, ccTLD
fo, ccTLD
fr, ccTLD
ga, ccTLD
gb, ccTLD
gd, ccTLD
ge, ccTLD
gf, ccTLD
gg, ccTLD
gh, ccTLD
gi, ccTLD
gl, ccTLD
gm, ccTLD
gn, ccTLD
gp, ccTLD
gq, ccTLD
gr, ccTLD
gs, ccTLD
gt, ccTLD
gu, ccTLD
gw, ccTLD
gy, ccTLD
hk, ccTLD
hm, ccTLD
hn, ccTLD
hr, ccTLD
ht, ccTLD
hu, ccTLD
id, ccTLD
ie, ccTLD
il, ccTLD
im, ccTLD
in, ccTLD
io, ccTLD
iq, ccTLD
ir, ccTLD
is, ccTLD
it, ccTLD
je, ccTLD
jm, ccTLD
jo, ccTLD
jp, ccTLD
ke, ccTLD
kg, ccTLD
kh, ccTLD
ki, ccTLD
km, ccTLD
kn, ccTLD
kp, ccTLD
kr, ccTLD
kw, ccTLD
ky, ccTLD
kz, ccTLD
la, ccTLD
lb, ccTLD
lc, ccTLD
li, ccTLD
lk, ccTLD
lr, ccTLD
ls, ccTLD
lt, ccTLD
lu, ccTLD
lv, ccTLD
ly, ccTLD
ma, ccTLD
mc, ccTLD
md, ccTLD
me, ccTLD
mg, ccTLD
mh, ccTLD
mk, ccTLD
ml, ccTLD
mm, ccTLD
mn, ccTLD
mo, ccTLD
mp, ccTLD
mq, ccTLD
mr, ccTLD
ms, ccTLD
mt, ccTLD
mu, ccTLD
mv, ccTLD
mw, ccTLD
mx, ccTLD
my, ccTLD
mz, ccTLD
na, ccTLD
nc, ccTLD
ne, ccTLD
nf, ccTLD
ng, ccTLD
ni, ccTLD
nl, ccTLD
no, ccTLD
np, ccTLD
nr, ccTLD
nu, ccTLD
nz, ccTLD
om, ccTLD
pa, ccTLD
pe, ccTLD
pf, ccTLD
pg, ccTLD
ph, ccTLD
pk, ccTLD
pl, ccTLD
pm, ccTLD
pn, ccTLD
pr, ccTLD
ps, ccTLD
pt, ccTLD
pw, ccTLD
py, ccTLD
qa, ccTLD
re, ccTLD
ro, ccTLD
rs, ccTLD
ru, ccTLD
rw, ccTLD
sa, ccTLD
sb, ccTLD
sc, ccTLD
sd, ccTLD
se, ccTLD
sg, ccTLD
sh, ccTLD
si, ccTLD
sj, ccTLD
sk, ccTLD
sl, ccTLD
sm, ccTLD
sn, ccTLD
so, ccTLD
sr, ccTLD
st, ccTLD
su, ccTLD
sv, ccTLD
sx, ccTLD
sy, ccTLD
sz, ccTLD
tc, ccTLD
td, ccTLD
tf, ccTLD
tg, ccTLD
th, ccTLD
tj, ccTLD
tk, ccTLD
tl, ccTLD
tm, ccTLD
tn, ccTLD
to, ccTLD
tr, ccTLD
tt, ccTLD
tv, ccTLD
tw, ccTLD
tz, ccTLD
ua, ccTLD
ug, ccTLD
uk, ccTLD
us, ccTLD
uy, ccTLD
uz, ccTLD
va, ccTLD
vc, ccTLD
ve, ccTLD
vg, ccTLD
vi, ccTLD
vn, ccTLD
vu, ccTLD
wf, ccTLD
ws, ccTLD
ye, ccTLD
yt, ccTLD
za, ccTLD
zm, ccTLD
zw, ccTLD
# Format is: alabel, type

xn--3e0b707e, ccTLD
xn--45brj9c, ccTLD
xn--80ao21a, ccTLD
xn--90a3ac, ccTLD
xn--90ais, ccTLD
xn--clchc0ea0b2g2a9gcd, ccTLD
xn--d1alf, ccTLD
xn--fiqs8s, ccTLD
xn--fiqz9s, ccTLD
xn--fpcrj9c3d, ccTLD
xn--fzc2c9e2c, ccTLD
xn--gecrj9c, ccTLD
xn--h2brj9c, ccTLD
xn--j1amh, ccTLD
xn--j6w193g, ccTLD
xn--kprw13d, ccTLD
xn--kpry57d, ccTLD
xn--l1acc, ccTLD
xn--lgbbat1ad8j, ccTLD
xn--mgb9awbf, ccTLD
xn--mgba3a4f16a, ccTLD
xn--mgbaam7a8h, ccTLD
xn--mgbayh7gpa, ccTLD
xn--mgbbh1a71e, ccTLD
xn--mgbc0a9azcg, ccTLD
xn--mgberp4a5d4ar, ccTLD
xn--mgbx4cd0ab, ccTLD
xn--node, ccTLD
xn--o3cw4h, ccTLD
xn--ogbpf8fl, ccTLD
xn--p1ai, ccTLD
xn--pgbs0dh, ccTLD
xn--s9brj9c, ccTLD
xn--wgbh1c, ccTLD
xn--wgbl6a, ccTLD
xn--xkc2al3hye2a, ccTLD
xn--xkc2dl3a5ee0h, ccTLD
xn--yfro4i67o, ccTLD
xn--ygbi2ammx, ccTLD
# Format is: alabel, type

aero, Legacy-gTLD
arpa, Legacy-gTLD
asia, Legacy-gTLD
biz, Legacy-gTLD
cat, Legacy-gTLD
com, Legacy-gTLD
coop, Legacy-gTLD
edu, Legacy-gTLD
gov, Legacy-gTLD
info, Legacy-gTLD
int, Legacy-gTLD
jobs, Legacy-gTLD
mil, Legacy-gTLD
mobi, Legacy-gTLD
museum, Legacy-gTLD
name, Legacy-gTLD
net, Legacy-gTLD
org, Legacy-gTLD
post, Legacy-gTLD
pro, Legacy-gTLD
sncf, Legacy-gTLD
tel, Legacy-gTLD
travel, Legacy-gTLD
xxx, Legacy-gTLD
# Format is: alabel, type

LASALLE, New-gTLD
JLC, New-gTLD
MICROSOFT, New-gTLD
HOTMAIL, New-gTLD
WINDOWS, New-gTLD
BING, New-gTLD
AZURE, New-gTLD
MONTBLANC, New-gTLD
HOMEDEPOT, New-gTLD
XBOX, New-gTLD
BIBLE, New-gTLD
SKI, New-gTLD
xn--imr513n, New-gTLD
WALTER, New-gTLD
SANDVIK, New-gTLD
SANDVIKCOROMANT, New-gTLD
BBVA, New-gTLD
MBA, New-gTLD
JLL, New-gTLD
FYI, New-gTLD
THD, New-gTLD
MEN, New-gTLD
LUPIN, New-gTLD
CORSICA, New-gTLD
CISCO, New-gTLD
EARTH, New-gTLD
SOCCER, New-gTLD
COUPONS, New-gTLD
ICBC, New-gTLD
BROTHER, New-gTLD
ACCENTURE, New-gTLD
PHILIPS, New-gTLD
xn--fjq720a, New-gTLD
NEC, New-gTLD
TAXI, New-gTLD
xn--estv75g, New-gTLD
HOCKEY, New-gTLD
RUN, New-gTLD
xn--y9a3aq, New-gTLD
xn--mgbpl2fh, New-gTLD
THEATER, New-gTLD
AIG, New-gTLD
ICU, New-gTLD
LOL, New-gTLD
LIAISON, New-gTLD
CFA, New-gTLD
NADEX, New-gTLD
CARS, New-gTLD
AUTO, New-gTLD
TORAY, New-gTLD
HITACHI, New-gTLD
BRIDGESTONE, New-gTLD
SENER, New-gTLD
RENT, New-gTLD
HONDA, New-gTLD
SWISS, New-gTLD
DOG, New-gTLD
BROKER, New-gTLD
ABB, New-gTLD
SEX, New-gTLD
SEAT, New-gTLD
WEIR, New-gTLD
JEWELRY, New-gTLD
XEROX, New-gTLD
SHOW, New-gTLD
TEAM, New-gTLD
SONY, New-gTLD
xn--kcrx77d1x4a, New-gTLD
EXPRESS, New-gTLD
BAUHAUS, New-gTLD
CAFE, New-gTLD
CYOU, New-gTLD
RACING, New-gTLD
xn--nyqy26a, New-gTLD
SCHOLARSHIPS, New-gTLD
LOVE, New-gTLD
xn--30rr7y, New-gTLD
MMA, New-gTLD
AFL, New-gTLD
REDSTONE, New-gTLD
xn--9et52u, New-gTLD
BOND, New-gTLD
SAP, New-gTLD
KOMATSU, New-gTLD
DOHA, New-gTLD
MTN, New-gTLD
REVIEW, New-gTLD
WIN, New-gTLD
PANERAI, New-gTLD
DOWNLOAD, New-gTLD
MOVIE, New-gTLD
FAITH, New-gTLD
TICKETS, New-gTLD
ACCOUNTANT, New-gTLD
LOAN, New-gTLD
DATE, New-gTLD
TOURS, New-gTLD
ADS, New-gTLD
PLUS, New-gTLD
FILM, New-gTLD
GOLD, New-gTLD
GOLF, New-gTLD
GUGE, New-gTLD
BBC, New-gTLD
TECH, New-gTLD
NEWS, New-gTLD
xn--vuq861b, New-gTLD
PAGE, New-gTLD
PIAGET, New-gTLD
SITE, New-gTLD
FAN, New-gTLD
ONLINE, New-gTLD
CFD, New-gTLD
TRADING, New-gTLD
SPREADBETTING, New-gTLD
ERNI, New-gTLD
MARKETS, New-gTLD
FOREX, New-gTLD
CHLOE, New-gTLD
XIN, New-gTLD
PICTET, New-gTLD
ABBOTT, New-gTLD
NISSAN, New-gTLD
INFINITI, New-gTLD
DATSUN, New-gTLD
MTPC, New-gTLD
ORACLE, New-gTLD
GOO, New-gTLD
LECLERC, New-gTLD
JAVA, New-gTLD
EPSON, New-gTLD
MAIF, New-gTLD
xn--mxtq1m, New-gTLD
SUCKS, New-gTLD
STUDY, New-gTLD
BOATS, New-gTLD
COURSES, New-gTLD
CASINO, New-gTLD
FANS, New-gTLD
FOOTBALL, New-gTLD
GOLDPOINT, New-gTLD
SCHOOL, New-gTLD
YODOBASHI, New-gTLD
CBN, New-gTLD
GDN, New-gTLD
APARTMENTS, New-gTLD
SAXO, New-gTLD
NICO, New-gTLD
TENNIS, New-gTLD
BINGO, New-gTLD
CANON, New-gTLD
STYLE, New-gTLD
TOSHIBA, New-gTLD
CHAT, New-gTLD
NTT, New-gTLD
KYOTO, New-gTLD
DCLK, New-gTLD
xn--b4w605ferd, New-gTLD
IFM, New-gTLD
GOOG, New-gTLD
TEMASEK, New-gTLD
HANGOUT, New-gTLD
BARCLAYS, New-gTLD
DESIGN, New-gTLD
DABUR, New-gTLD
HERMES, New-gTLD
BARCLAYCARD, New-gTLD
JCB, New-gTLD
ONE, New-gTLD
LOTTE, New-gTLD
MARRIOTT, New-gTLD
FIT, New-gTLD
LAT, New-gTLD
KDDI, New-gTLD
BANK, New-gTLD
SHRIRAM, New-gTLD
ZUERICH, New-gTLD
FLOWERS, New-gTLD
GGEE, New-gTLD
SALE, New-gTLD
AMSTERDAM, New-gTLD
VIDEO, New-gTLD
DOCS, New-gTLD
TIRES, New-gTLD
DEV, New-gTLD
SCHWARZ, New-gTLD
IWC, New-gTLD
OSAKA, New-gTLD
SEW, New-gTLD
GARDEN, New-gTLD
LIDL, New-gTLD
DOOSAN, New-gTLD
SKY, New-gTLD
CARTIER, New-gTLD
SAMSUNG, New-gTLD
xn--czrs0t, New-gTLD
ADULT, New-gTLD
PORN, New-gTLD
TRUST, New-gTLD
EUROVISION, New-gTLD
FASHION, New-gTLD
LATROBE, New-gTLD
IRISH, New-gTLD
xn--hxt814e, New-gTLD
AQUARELLE, New-gTLD
MEMORIAL, New-gTLD
LEGAL, New-gTLD
MONEY, New-gTLD
COACH, New-gTLD
EVERBANK, New-gTLD
xn--flw351e, New-gTLD
MADRID, New-gTLD
FIRMDALE, New-gTLD
xn--qcka1pmc, New-gTLD
MORMON, New-gTLD
LDS, New-gTLD
PARTY, New-gTLD
xn--45q11c, New-gTLD
CRICKET, New-gTLD
SCIENCE, New-gTLD
REIT, New-gTLD
ANDROID, New-gTLD
SYDNEY, New-gTLD
BLOOMBERG, New-gTLD
ENERGY, New-gTLD
DELIVERY, New-gTLD
TAIPEI, New-gTLD
EMERCK, New-gTLD
FLSMIDTH, New-gTLD
BAND, New-gTLD
YOGA, New-gTLD
CRS, New-gTLD
ABOGADO, New-gTLD
RIP, New-gTLD
WEDDING, New-gTLD
POKER, New-gTLD
ALSACE, New-gTLD
ALLFINANZ, New-gTLD
IBM, New-gTLD
FORSALE, New-gTLD
xn--vermgensberatung-pwb, New-gTLD
xn--vermgensberater-ctb, New-gTLD
xn--p1acf, New-gTLD
TUI, New-gTLD
DVAG, New-gTLD
POHL, New-gTLD
WORK, New-gTLD
CASA, New-gTLD
BUDAPEST, New-gTLD
WORLD, New-gTLD
GOOGLE, New-gTLD
FLY, New-gTLD
NEXUS, New-gTLD
CHANNEL, New-gTLD
PROF, New-gTLD
GLE, New-gTLD
ZIP, New-gTLD
CAL, New-gTLD
CHROME, New-gTLD
WME, New-gTLD
PHARMACY, New-gTLD
GMX, New-gTLD
BOO, New-gTLD
DAD, New-gTLD
DAY, New-gTLD
FRL, New-gTLD
ING, New-gTLD
NEW, New-gTLD
MOV, New-gTLD
RSVP, New-gTLD
EAT, New-gTLD
MEME, New-gTLD
YOUTUBE, New-gTLD
HERE, New-gTLD
PROD, New-gTLD
ESQ, New-gTLD
IMMO, New-gTLD
PIZZA, New-gTLD
GMAIL, New-gTLD
GBIZ, New-gTLD
OTSUKA, New-gTLD
BUSINESS, New-gTLD
xn--vhquv, New-gTLD
NETWORK, New-gTLD
CLICK, New-gTLD
DIET, New-gTLD
HOW, New-gTLD
OOO, New-gTLD
UOL, New-gTLD
HELP, New-gTLD
HOSTING, New-gTLD
PROPERTY, New-gTLD
LTDA, New-gTLD
CERN, New-gTLD
CARAVAN, New-gTLD
SCA, New-gTLD
xn--xhq521b, New-gTLD
xn--1qqw23a, New-gTLD
BNPPARIBAS, New-gTLD
RESTAURANT, New-gTLD
CYMRU, New-gTLD
SARL, New-gTLD
GIFTS, New-gTLD
WALES, New-gTLD
TATAR, New-gTLD
TOP, New-gTLD
HEALTHCARE, New-gTLD
REALTOR, New-gTLD
WILLIAMHILL, New-gTLD
ONG, New-gTLD
PRAXI, New-gTLD
YANDEX, New-gTLD
AUCTION, New-gTLD
WHOSWHO, New-gTLD
KRD, New-gTLD
NGO, New-gTLD
NRA, New-gTLD
SPIEGEL, New-gTLD
LACAIXA, New-gTLD
LGBT, New-gTLD
GENT, New-gTLD
NRW, New-gTLD
SCB, New-gTLD
DEALS, New-gTLD
MELBOURNE, New-gTLD
CITY, New-gTLD
SCHMIDT, New-gTLD
CUISINELLA, New-gTLD
CANCERRESEARCH, New-gTLD
DIRECT, New-gTLD
PLACE, New-gTLD
SUZUKI, New-gTLD
ACTIVE, New-gTLD
MINI, New-gTLD
BMW, New-gTLD
OVH, New-gTLD
CAPETOWN, New-gTLD
PHYSIO, New-gTLD
LOTTO, New-gTLD
DURBAN, New-gTLD
GREEN, New-gTLD
JOBURG, New-gTLD
VLAANDEREN, New-gTLD
BRUSSELS, New-gTLD
SURF, New-gTLD
xn--kput3i, New-gTLD
BZH, New-gTLD
SCOT, New-gTLD
ORGANIC, New-gTLD
GLOBAL, New-gTLD
ENGINEER, New-gTLD
NHK, New-gTLD
HAMBURG, New-gTLD
REHAB, New-gTLD
TIROL, New-gTLD
REPUBLICAN, New-gTLD
GIVES, New-gTLD
NAVY, New-gTLD
ARMY, New-gTLD
BIO, New-gTLD
LAWYER, New-gTLD
VET, New-gTLD
MORTGAGE, New-gTLD
SOFTWARE, New-gTLD
HOST, New-gTLD
MARKET, New-gTLD
DENTIST, New-gTLD
HIV, New-gTLD
ATTORNEY, New-gTLD
PRESS, New-gTLD
DEGREE, New-gTLD
SPACE, New-gTLD
WEBSITE, New-gTLD
xn--4gbrim, New-gTLD
YACHTS, New-gTLD
RIO, New-gTLD
HOMES, New-gTLD
AUTOS, New-gTLD
VERSICHERUNG, New-gTLD
REISE, New-gTLD
MOTORCYCLES, New-gTLD
xn--czr694b, New-gTLD
GUIDE, New-gTLD
AUDIO, New-gTLD
HIPHOP, New-gTLD
LOANS, New-gTLD
CHURCH, New-gTLD
LIFE, New-gTLD
BEER, New-gTLD
JUEGOS, New-gTLD
LUXE, New-gTLD
CREDIT, New-gTLD
ACCOUNTANTS, New-gTLD
DIGITAL, New-gTLD
CLAIMS, New-gTLD
GMO, New-gTLD
GLOBO, New-gTLD
BAYERN, New-gTLD
AIRFORCE, New-gTLD
CREDITCARD, New-gTLD
INSURE, New-gTLD
FINANCE, New-gTLD
WTC, New-gTLD
CITIC, New-gTLD
xn--80adxhks, New-gTLD
MOSCOW, New-gTLD
LIMITED, New-gTLD
FAIL, New-gTLD
EXCHANGE, New-gTLD
TAX, New-gTLD
WTF, New-gTLD
FUND, New-gTLD
SURGERY, New-gTLD
INVESTMENTS, New-gTLD
FINANCIAL, New-gTLD
GRATIS, New-gTLD
FURNITURE, New-gTLD
DENTAL, New-gTLD
CARE, New-gTLD
CASH, New-gTLD
DISCOUNT, New-gTLD
BLACKFRIDAY, New-gTLD
CLINIC, New-gTLD
FITNESS, New-gTLD
SCHULE, New-gTLD
FOO, New-gTLD
SOY, New-gTLD
PARIS, New-gTLD
FROGANS, New-gTLD
QUEBEC, New-gTLD
PICTURES, New-gTLD
UNIVERSITY, New-gTLD
GAL, New-gTLD
ASSOCIATES, New-gTLD
REISEN, New-gTLD
MEDIA, New-gTLD
CAREER, New-gTLD
TOWN, New-gTLD
TOYS, New-gTLD
LEASE, New-gTLD
SERVICES, New-gTLD
ENGINEERING, New-gTLD
GRIPE, New-gTLD
EUS, New-gTLD
CAPITAL, New-gTLD
DESI, New-gTLD
FEEDBACK, New-gTLD
COLLEGE, New-gTLD
ROCKS, New-gTLD
xn--ses554g, New-gTLD
GOP, New-gTLD
RYUKYU, New-gTLD
YOKOHAMA, New-gTLD
REST, New-gTLD
SAARLAND, New-gTLD
CONSULTING, New-gTLD
VODKA, New-gTLD
HAUS, New-gTLD
COOKING, New-gTLD
MOE, New-gTLD
RODEO, New-gTLD
COUNTRY, New-gTLD
xn--czru2d, New-gTLD
HORSE, New-gTLD
FISHING, New-gTLD
VEGAS, New-gTLD
MIAMI, New-gTLD
ARCHI, New-gTLD
BLACK, New-gTLD
REN, New-gTLD
MEET, New-gTLD
SOHU, New-gTLD
LONDON, New-gTLD
NYC, New-gTLD
COLOGNE, New-gTLD
AXA, New-gTLD
WEBCAM, New-gTLD
TRADE, New-gTLD
JETZT, New-gTLD
xn--rhqv96g, New-gTLD
DNP, New-gTLD
INK, New-gTLD
xn--nqv7f, New-gTLD
xn--i1b6b1a6a2e, New-gTLD
xn--nqv7fs00ema, New-gTLD
xn--c1avg, New-gTLD
KOELN, New-gTLD
BID, New-gTLD
OKINAWA, New-gTLD
VOTE, New-gTLD
VOTO, New-gTLD
BAR, New-gTLD
KRED, New-gTLD
BEST, New-gTLD
CHRISTMAS, New-gTLD
PUB, New-gTLD
xn--d1acj3b, New-gTLD
ACTOR, New-gTLD
SUPPLIES, New-gTLD
FISH, New-gTLD
xn--cg4bki, New-gTLD
VACATIONS, New-gTLD
INDUSTRIES, New-gTLD
SUPPLY, New-gTLD
XYZ, New-gTLD
WIKI, New-gTLD
NEUSTAR, New-gTLD
xn--mgbab2bd, New-gTLD
MANGO, New-gTLD
QPON, New-gTLD
CARDS, New-gTLD
FOUNDATION, New-gTLD
REVIEWS, New-gTLD
FUTBOL, New-gTLD
VISION, New-gTLD
CONDOS, New-gTLD
VILLAS, New-gTLD
PARTS, New-gTLD
PRODUCTIONS, New-gTLD
MAISON, New-gTLD
xn--6frz82g, New-gTLD
BLUE, New-gTLD
FLIGHTS, New-gTLD
CLEANING, New-gTLD
REPORT, New-gTLD
EVENTS, New-gTLD
CRUISES, New-gTLD
RENTALS, New-gTLD
PARTNERS, New-gTLD
PROPERTIES, New-gTLD
CATERING, New-gTLD
EXPOSED, New-gTLD
VOTING, New-gTLD
TOKYO, New-gTLD
NAGOYA, New-gTLD
DATING, New-gTLD
COMMUNITY, New-gTLD
BOUTIQUE, New-gTLD
BARGAINS, New-gTLD
TIENDA, New-gTLD
WORKS, New-gTLD
WED, New-gTLD
EXPERT, New-gTLD
WATCH, New-gTLD
KIM, New-gTLD
COOL, New-gTLD
TOOLS, New-gTLD
CLUB, New-gTLD
BUILD, New-gTLD
PICS, New-gTLD
PINK, New-gTLD
LUXURY, New-gTLD
PHOTO, New-gTLD
GIFT, New-gTLD
xn--io0a7i, New-gTLD
xn--55qx5d, New-gTLD
MONASH, New-gTLD
RICH, New-gTLD
RED, New-gTLD
SHIKSHA, New-gTLD
xn--fiq64b, New-gTLD
LINK, New-gTLD
GUITARS, New-gTLD
ZONE, New-gTLD
CHEAP, New-gTLD
MARKETING, New-gTLD
DEMOCRAT, New-gTLD
SOCIAL, New-gTLD
AGENCY, New-gTLD
MODA, New-gTLD
DANCE, New-gTLD
BERLIN, New-gTLD
xn--3bst00m, New-gTLD
xn--6qq986b3xl, New-gTLD
xn--fiq228c5hs, New-gTLD
WANG, New-gTLD
KIWI, New-gTLD
WIEN, New-gTLD
EMAIL, New-gTLD
IMMOBILIEN, New-gTLD
xn--3ds443g, New-gTLD
INSTITUTE, New-gTLD
CEO, New-gTLD
SOLAR, New-gTLD
FARM, New-gTLD
EDUCATION, New-gTLD
GLASS, New-gTLD
ONL, New-gTLD
INTERNATIONAL, New-gTLD
CODES, New-gTLD
TRAINING, New-gTLD
HOUSE, New-gTLD
KAUFEN, New-gTLD
NINJA, New-gTLD
REPAIR, New-gTLD
BUILDERS, New-gTLD
COFFEE, New-gTLD
FLORIST, New-gTLD
HOLIDAY, New-gTLD
SOLUTIONS, New-gTLD
BUZZ, New-gTLD
SUPPORT, New-gTLD
RECIPES, New-gTLD
COMPUTER, New-gTLD
ACADEMY, New-gTLD
CAREERS, New-gTLD
CAB, New-gTLD
xn--55qw42g, New-gTLD
xn--zfr164b, New-gTLD
SYSTEMS, New-gTLD
DOMAINS, New-gTLD
VIAJES, New-gTLD
COMPANY, New-gTLD
CAMP, New-gTLD
LIMO, New-gTLD
MANAGEMENT, New-gTLD
PHOTOS, New-gTLD
SHOES, New-gTLD
CENTER, New-gTLD
RUHR, New-gTLD
MENU, New-gTLD
UNO, New-gTLD
xn--q9jyb4c, New-gTLD
DIAMONDS, New-gTLD
TIPS, New-gTLD
PHOTOGRAPHY, New-gTLD
DIRECTORY, New-gTLD
ENTERPRISES, New-gTLD
KITCHEN, New-gTLD
TODAY, New-gTLD
PLUMBING, New-gTLD
GRAPHICS, New-gTLD
CONTRACTORS, New-gTLD
GALLERY, New-gTLD
SEXY, New-gTLD
CONSTRUCTION, New-gTLD
TATTOO, New-gTLD
TECHNOLOGY, New-gTLD
ESTATE, New-gTLD
LAND, New-gTLD
BIKE, New-gTLD
VENTURES, New-gTLD
CAMERA, New-gTLD
CLOTHING, New-gTLD
LIGHTING, New-gTLD
SINGLES, New-gTLD
VOYAGE, New-gTLD
GURU, New-gTLD
HOLDINGS, New-gTLD
EQUIPMENT, New-gTLD
xn--ngbc5azd, New-gTLD
xn--80asehdb, New-gTLD
xn--80aswg, New-gTLD
xn--unup4y, New-gTLD);
# Format is: alabel, type

MOBILY, New-gTLD
xn--mgbb9fbpob, New-gTLD
STAR, New-gTLD
PID, New-gTLD
CONTACT, New-gTLD
WEBER, New-gTLD
ANALYTICS, New-gTLD
JMP, New-gTLD
xn--jlq61u9w7b, New-gTLD
SAS, New-gTLD
FORD, New-gTLD
STORAGE, New-gTLD
LINCOLN, New-gTLD
KFH, New-gTLD
WANGGOU, New-gTLD
SHELL, New-gTLD
xn--pbt977c, New-gTLD
xn--kpu716f, New-gTLD
KPN, New-gTLD
xn--ngbe9e0a, New-gTLD
TRAVELERSINSURANCE, New-gTLD
WATCHES, New-gTLD
xn--eckvdtc9d, New-gTLD
TUSHU, New-gTLD
TRV, New-gTLD
REDUMBRELLA, New-gTLD
xn--mgbt3dhd, New-gTLD
PARS, New-gTLD
ZERO, New-gTLD
SMILE, New-gTLD
BOT, New-gTLD
JOY, New-gTLD
BUY, New-gTLD
GOT, New-gTLD
FAST, New-gTLD
JOT, New-gTLD
PIN, New-gTLD
READ, New-gTLD
CIRCLE, New-gTLD
BOOK, New-gTLD
SAFE, New-gTLD
CALL, New-gTLD
ROOM, New-gTLD
LIKE, New-gTLD
AUTHOR, New-gTLD
TRAVELERS, New-gTLD
NOWRUZ, New-gTLD
FIRESTONE, New-gTLD
TCI, New-gTLD
SHARP, New-gTLD
SALON, New-gTLD
SHIA, New-gTLD
INSURANCE, New-gTLD
NORTON, New-gTLD
MED, New-gTLD
SYMANTEC, New-gTLD
SFR, New-gTLD
BOEHRINGER, New-gTLD
BOSTIK, New-gTLD
LAMBORGHINI, New-gTLD
BUGATTI, New-gTLD
AUDI, New-gTLD
VIP, New-gTLD
VERISIGN, New-gTLD
BROADWAY, New-gTLD
COMSEC, New-gTLD
GRAINGER, New-gTLD
TAB, New-gTLD
FAIRWINDS, New-gTLD
CITYEATS, New-gTLD
LIFESTYLE, New-gTLD
CREDITUNION, New-gTLD
VANA, New-gTLD
PLAYSTATION, New-gTLD
FERRERO, New-gTLD
ROCHER, New-gTLD
APPLE, New-gTLD
AARP, New-gTLD
BEATS, New-gTLD
PING, New-gTLD
MEO, New-gTLD
SAPO, New-gTLD
SBS, New-gTLD
GUCCI, New-gTLD
LANDROVER, New-gTLD
JAGUAR, New-gTLD
RWE, New-gTLD
ZARA, New-gTLD
MUTUELLE, New-gTLD
ARTE, New-gTLD
ARAMCO, New-gTLD
xn--mgba3a3ejt, New-gTLD
DELL, New-gTLD
CIPRIANI, New-gTLD
KINDER, New-gTLD
YAMAXUN, New-gTLD
MOI, New-gTLD
VIRGIN, New-gTLD
MTR, New-gTLD
CLUBMED, New-gTLD
BOM, New-gTLD
STOCKHOLM, New-gTLD
KIA, New-gTLD
FINAL, New-gTLD
SEVEN, New-gTLD
HYUNDAI, New-gTLD
OBI, New-gTLD
LTD, New-gTLD
BMS, New-gTLD
SECURITY, New-gTLD
LINDE, New-gTLD
STADA, New-gTLD
THEATRE, New-gTLD
PROTECTION, New-gTLD
CAR, New-gTLD
CSC, New-gTLD
AMICA, New-gTLD
STC, New-gTLD
VIVA, New-gTLD
STCGROUP, New-gTLD
AAA, New-gTLD
GEA, New-gTLD
ACO, New-gTLD
xn--efvy88h, New-gTLD
MOM, New-gTLD
FAMILY, New-gTLD
SEEK, New-gTLD
FAGE, New-gTLD
CEB, New-gTLD
GROUP, New-gTLD
GIVING, New-gTLD
XPERIA, New-gTLD
BOOTS, New-gTLD
CHANEL, New-gTLD
WINE, New-gTLD
VIN, New-gTLD
LIXIL, New-gTLD
xn--tckwe, New-gTLD
xn--fhbei, New-gTLD
xn--c2br7g, New-gTLD
xn--9dbq2a, New-gTLD
xn--11b4c3d, New-gTLD
xn--3pxu8k, New-gTLD
xn--t60b56a, New-gTLD
xn--j1aef, New-gTLD
xn--pssy2u, New-gTLD
xn--42c2d9a, New-gTLD
xn--mk1bu44c, New-gTLD
IPIRANGA, New-gTLD
LEXUS, New-gTLD
PET, New-gTLD
MAN, New-gTLD
TOYOTA, New-gTLD
SANOFI, New-gTLD
TATAMOTORS, New-gTLD
SRL, New-gTLD
BET, New-gTLD
ICE, New-gTLD
ITAU, New-gTLD
NOKIA, New-gTLD
LANCASTER, New-gTLD
UBS, New-gTLD
ISTANBUL, New-gTLD
DELTA, New-gTLD
IST, New-gTLD
HSBC, New-gTLD
IINET, New-gTLD
ORANGE, New-gTLD
BENTLEY, New-gTLD
GAME, New-gTLD
JPRS, New-gTLD
BCN, New-gTLD
STUDIO, New-gTLD
BARCELONA, New-gTLD
AIRTEL, New-gTLD
LIVE, New-gTLD
APP, New-gTLD
SAKURA, New-gTLD
REALTY, New-gTLD
FORUM, New-gTLD
HOTELES, New-gTLD
TELEFONICA, New-gTLD
BNL, New-gTLD
CLOUD, New-gTLD
BRADESCO, New-gTLD
MOVISTAR, New-gTLD
OMEGA, New-gTLD
SWATCH, New-gTLD
LAW, New-gTLD
SKYPE, New-gTLD
SCOR, New-gTLD
OFFICE, New-gTLD
STARHUB, New-gTLD
VISTA, New-gTLD
COMMBANK, New-gTLD
VISTAPRINT, New-gTLD
RICOH, New-gTLD
NETBANK, New-gTLD
CBA, New-gTLD
GENTING, New-gTLD
DRIVE, New-gTLD
PLAY, New-gTLD
AEG, New-gTLD
CROWN, New-gTLD
STATOIL, New-gTLD
BHARTI, New-gTLD
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#

#                                               -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_PREREQ(2.61)
AC_INIT(Hedgehog, 2.4.2,  dns-stats-users@dns-stats.org)
AC_CONFIG_SRCDIR([tools/hedgehogctl.in])
AC_CONFIG_HEADER([config.h])

AC_DEFINE_UNQUOTED([DB_SCHEMA_VERSION],[14],[Database version])
AC_SUBST([DB_SCHEMA_VERSION],[14])
AC_SUBST([VERSION_NAME],[Bare-bellied])

AC_CONFIG_MACRO_DIR([m4])

AM_INIT_AUTOMAKE([foreign])
m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])

# Checks for programs.
AC_PROG_CC
AC_PROG_CPP
AC_PROG_CXX
AC_PROG_AWK
AC_PROG_SED
AC_PROG_MAKE_SET
AC_PROG_INSTALL
AC_PROG_LN_S
AC_PROG_LIBTOOL
LT_INIT
AM_PATH_PYTHON
AC_PATH_PROG(HELP2MAN, help2man)
if test -z "$HELP2MAN"
then
  AC_MSG_ERROR("help2man not found")
fi

AC_CONFIG_FILES(Makefile conf/Makefile tools/Makefile)

WEB=1
DM=1
ACX_COMPONENTS
AS_IF([test $WEB -eq 1],[AC_MSG_NOTICE([Building Web Component...])
  AC_CONFIG_FILES(www/Makefile)

],[AC_MSG_NOTICE([Not building Web Component...])])

AS_IF([test $DM -eq 1],[AC_MSG_NOTICE([Building Data Manager Component...])

  # Checks for header files.
  AC_CHECK_HEADERS([arpa/inet.h fcntl.h netinet/in.h stdlib.h string.h sys/param.h unistd.h])

  # Checks for typedefs, structures, and compiler characteristics.
  AC_C_CONST
  AC_STRUCT_TM
  AC_TYPE_INT16_T
  AC_TYPE_INT32_T
  AC_TYPE_INT64_T
  AC_TYPE_INT8_T
  AC_TYPE_PID_T
  AC_TYPE_SIZE_T
  AC_TYPE_UINT16_T
  AC_TYPE_UINT32_T
  AC_TYPE_UINT64_T
  AC_TYPE_UINT8_T
  # Checks for library functions.
  AC_FUNC_STRFTIME
  AC_FUNC_MALLOC
  AC_FUNC_MKTIME
  AC_FUNC_REALLOC
  AC_CHECK_FUNCS([gettimeofday mempcpy memset strerror])

  BOOST_REQUIRE([1.41.0])
  BOOST_SYSTEM
  BOOST_FILESYSTEM
  BOOST_PROGRAM_OPTIONS
  BOOST_CONVERSION
  BOOST_UNORDERED
  BOOST_FOREACH
  BOOST_STRING_ALGO
  BOOST_TOKENIZER
  
  # Checks for libraries.
  ACX_POSTGRES
  ACX_LIBPQXX

  CXXFLAGS="${CXXFLAGS} -Wall -pedantic \
           -pedantic-errors -Wextra"

  AC_CONFIG_FILES(src/Makefile)
  
],[AC_MSG_NOTICE([Not building Data Manager Component...])])

AM_CONDITIONAL([DM], [test $DM -eq 1])
AM_CONDITIONAL([WEB], [test $WEB -eq 1])

AC_OUTPUT
%PDF-1.4
%âãÏÓ
4 0 obj
<</Filter/FlateDecode/Length 301>>stream
xËNÃ0E÷ó³qb§IX"ñ"#'ÒÄ´¿Ã§bG}JHÈ²4wlß9WîáNÇLF,ËP¸W°8ã"Mp1>ûóD_àã£ap¿D¦,>êõAËqé¥¼(·jxÎÛåªpóa³U	ÑÜP¦-ª®ÈTTÛ
ß6ÓðqjÝbÌ$p´s_«U1{Æ²<¿ðôB0Ìªn4¶ZêF44®ù¢k»~Ó~ÆÜÎ¶èhMz ÓØóF¶Ä#è¶Ñ¸ÑÅ·®h`?çd>­ø+-ç3°g{=wý7Ô"ÍUMÎ3ûÝÙ.ÆÆví§ÆÁÒ:Cþ}J¶~	¿dþø
endstream
endobj
1 0 obj
<</Contents 4 0 R/Type/Page/Resources<</ProcSet [/PDF /Text /ImageB /ImageC /ImageI]/Font<</F1 2 0 R/F2 3 0 R>>>>/Parent 5 0 R/MediaBox[0 0 595.28 841.88]>>
endobj
7 0 obj
<</Dest[1 0 R/XYZ 0 793.88 0]/Title(Hedgehog Upgrade Guide: 2.4.1 to 2.4.2)/Parent 6 0 R>>
endobj
6 0 obj
<</Type/Outlines/Count 1/First 7 0 R/Last 7 0 R>>
endobj
2 0 obj
<</Subtype/Type1/Type/Font/BaseFont/Helvetica-Bold/Encoding/WinAnsiEncoding>>
endobj
3 0 obj
<</Subtype/Type1/Type/Font/BaseFont/Helvetica/Encoding/WinAnsiEncoding>>
endobj
5 0 obj
<</Kids[1 0 R]/Type/Pages/Count 1/ITXT(2.1.7)>>
endobj
8 0 obj
<</PageMode/UseOutlines/Type/Catalog/Outlines 6 0 R/Pages 5 0 R>>
endobj
9 0 obj
<</ModDate(D:20190108144708Z)/CreationDate(D:20190108144708Z)/Producer(iText 2.1.7 by 1T3XT)>>
endobj
xref
0 10
0000000000 65535 f 
0000000383 00000 n 
0000000726 00000 n 
0000000819 00000 n 
0000000015 00000 n 
0000000907 00000 n 
0000000661 00000 n 
0000000555 00000 n 
0000000970 00000 n 
0000001051 00000 n 
trailer
<</Info 9 0 R/ID [<a7d465f00f59a80196f86ed6bf292470><d65fc21bb7a817f5f06274abf2f3158e>]/Root 8 0 R/Size 10>>
startxref
1161
%%EOF

<p><ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="f9e02cbb-e6f6-4fe6-b9b0-812aded99f27" /></p>
<p>The following instructions describe how to install Hedgehog from Ubuntu Packages on Ubuntu 14.04 Server and set up imports of data.</p>
<p><br /></p><ac:structured-macro ac:name="numberedheadings" ac:schema-version="1" ac:macro-id="b97e1073-8939-41e4-9abc-f56b74e227db"><ac:rich-text-body>
<h1>Requirements</h1>
<h2>Platform support</h2>
<p>Hedgehog is currently only supported on Ubuntu 14.04 LTS Server.</p>
<h2>Database support</h2>
<p>Hedgehog has been tested with PostgreSQL 9.3.X and 9.6.X. The installation of the database is not covered in detail since it may or may not be co-located with the other hedgehog components.</p>
<p>The <span>hedgehog-database package (see below) </span>will suggest the following two packages:</p>
<ul>
<li>
<p>postgresql-9.3-ip4r</p></li>
<li>
<p>postgresql-plpython-9.3</p></li></ul>
<p>Versions of these packages appropriate to the version of PostgreSQL installed are required.</p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="d352c3b5-d0f4-4f56-93a5-e788432f549f"><ac:rich-text-body>
<p>The Hedgehog servers and PostgreSQL must both be configured to use UTC!</p></ac:rich-text-body></ac:structured-macro>
<h2>General</h2>
<h3>Home directory permissions</h3>
<p>Some users set the permissions on their home directory such that other users cannot read or execute that directory. In this case when running scripts that update the database as the <em>hedgehog</em> or <em>postgres</em> (user via 'sudo -u &lt;user&gt;') meaningless <span>messages are generated from psql about being unable to cd into logged in users home directory. To avoid this</span> spurious output change directory to one that allows other users to read/execute (e.g. /tmp) and run the scripts from there.</p>
<h1>Hedgehog Installation</h1>
<h2>Hedgehog packages</h2>
<p>The Hedgehog PPA is located at</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="3aec0f80-ac4a-4605-8897-f99062b8fcef"><ac:plain-text-body><![CDATA[sudo add-apt-repository ppa:dns-stats/presenter]]></ac:plain-text-body></ac:structured-macro>
<p>Hedgehog comprises 3 main components which can be run on the same or different machines. Configure the additional PPA's as required and then install the main packages from the list below:</p>
<table class="wrapped"><colgroup><col /><col /><col /><col /><col /></colgroup>
<tbody>
<tr>
<th>Component</th>
<th>Description</th>
<th><span>Main Package</span></th>
<th colspan="1">Optional Package</th>
<th colspan="1">Additional Required PPAs</th></tr>
<tr>
<td colspan="1">Database</td>
<td colspan="1">PostgreSQL database for Hedgehog</td>
<td colspan="1">hedgehog-database</td>
<td colspan="1"><br /></td>
<td colspan="1">
<p>sudo add-apt-repository ppa:ondrej/pkg-nlnetlabs</p></td></tr>
<tr>
<td rowspan="2"><span>Data Manager</span></td>
<td>
<p>Scripts for database population and management (including rssacd)</p>
<p>Scripts for XML/DAT processing</p></td>
<td><span>hedgehog-data-manager</span></td>
<td colspan="1"><br /></td>
<td colspan="1"><br /></td></tr>
<tr>
<td colspan="1">(Optional) Apache config for webdav based XML uploads</td>
<td colspan="1"><br /></td>
<td colspan="1"><span>hedgehog-webdav-upload</span></td>
<td colspan="1"><br /></td></tr>
<tr>
<td>Web GUI</td>
<td>Web GUI front-end</td>
<td><span>hedgehog-gui</span></td>
<td colspan="1"><br /></td>
<td colspan="1">
<p>sudo add-apt-repository ppa:opencpu/rapache</p></td></tr></tbody></table>
<p>Notes:</p>
<ul>
<li>Multiple Web GUI's can be connected to the same database.</li>
<li>The upload mechanism for XML/DAT files is for the user to decide (ssh, rsync, Apache, etc.). A package for upload using Apache is provided for convenience.</li></ul>
<p>For reference the other Hedgehog packages are:</p>
<table class="wrapped">
<tbody>
<tr>
<th>Description</th>
<th>Required by</th>
<th><span>Package</span></th></tr>
<tr>
<td>Common package</td>
<td><span>All main hedgehog packages</span></td>
<td>hedgehog-common</td></tr>
<tr>
<td colspan="1">RPostgresHelper</td>
<td colspan="1">Web Gui</td>
<td colspan="1">r-hedgehog-rpostgresqlhelper</td></tr>
<tr>
<td colspan="1">Documentation</td>
<td colspan="1"><br /></td>
<td colspan="1">hedgehog-doc</td></tr></tbody></table>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="30c0ff9d-f3ef-44b1-9d42-b14508dddb31"><ac:rich-text-body>
<p><span>Two system users are used by Hedgehog, which are created by the packages:</span></p>
<table class="wrapped">
<tbody>
<tr>
<th>User</th>
<th>Default</th>
<th colspan="1">Component</th>
<th>Note</th>
<th colspan="1">Created by</th></tr>
<tr>
<td>Database owner</td>
<td>hedgehog</td>
<td colspan="1">
<p>Data Manager</p></td>
<td>
<p>This is the user that will own the database created by</p>
<p>Hedgehog and the top level datafile directories.</p></td>
<td colspan="1">hedgehog-common package</td></tr>
<tr>
<td>Database read user</td>
<td>www-data</td>
<td colspan="1">
<p>Web front-end</p></td>
<td>
<p>This defaults to the same as the default apache user</p></td>
<td colspan="1">apache2 package installation</td></tr></tbody></table>
<p><span><br /></span></p></ac:rich-text-body></ac:structured-macro>
<p><br /></p>
<h2>Web GUI only - Manual install of R packages</h2>
<p>These instructions&nbsp;are for R packages that must be installed using R's built-in compilation tools (there is no Ubuntu package for them).</p>
<p>To install interactively: (Some can repositories don't contain packages for all versions of R so if this fails quit R and re-try a different repo)</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="41d72df2-2e8d-49a1-b22e-ea5313c97f16"><ac:plain-text-body><![CDATA[sudo R
install.packages(c("brew","Cairo","googleVis","R.utils","yaml","plyr"))
q()
# If you are prompted to save workspace image y/n/c, choose no. ]]></ac:plain-text-body></ac:structured-macro>
<p><span>For scripting purposes a repo can be specified by using a command of the form:</span></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="e85ac8e9-5693-47be-bf61-a712d38cd191"><ac:plain-text-body><![CDATA[install.packages("name", repos='http://cran.rstudio.com/')]]></ac:plain-text-body></ac:structured-macro>
<p><span>Hedgehog is tested against version 3.1.1 of R.&nbsp;</span>Hedgehog requires at least the versions below of each package.</p>
<table class="wrapped">
<thead>
<tr>
<th>
<div class="tablesorter-header-inner">R Package</div></th>
<th>
<div class="tablesorter-header-inner">Supported Version</div></th></tr></thead>
<tbody>
<tr>
<td>brew</td>
<td>1.0-6</td></tr>
<tr>
<td>Cairo</td>
<td>1.5-9</td></tr>
<tr>
<td>googleVis</td>
<td>
<p>0.5.10</p></td></tr>
<tr>
<td colspan="1">R.utils</td>
<td colspan="1">2.2.0</td></tr>
<tr>
<td colspan="1">yaml</td>
<td colspan="1">2.1.13</td></tr>
<tr>
<td colspan="1">plyr</td>
<td colspan="1">
<p class="p1"><span class="s1">1.8.3</span></p></td></tr></tbody></table>
<p><br /></p>
<h1>Hedgehog configuration</h1>
<ul>
<li>For the Web GUI and Data Manager components, ensure the /etc/hedgehog/hedgehog.yaml configuration file contains usernames and passwords that will match those configured in the database (see next section).</li>
<li>Also configure the database parameters (host, port and name) as required.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="b90d6358-dbe9-4d83-bfe3-f3addfa11f8e"><ac:plain-text-body><![CDATA[database: 
  host         : /var/run/postgresql # specify a host for the postgresql DB. If
                                     # this begins with a slash, it specifies the
                                     # directory in which the socket file is 
                                     # stored.
  port         : 5432      # specify port for the postgresql DB.
  name         : hedgehog  # specify dbname for the postgresql DB.
  owner        : hedgehog  # specify a user to own the postgresql DB. 
                           # [Required for Data Manager component]
  owner_pass   :           # specify a password for the owner user if needed.
  read_user    : www-data  # specify a read user for the postgresql DB. 
                           # [Required for Web front-end component]
  read_pass    :           # specify a password for the read user if needed.
]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="67324d78-e7f4-4511-9743-e5f797434297"><ac:rich-text-body>
<p><span>Do not edit the 'directories' section of this file as it is auto-generated.</span></p></ac:rich-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="5be45376-3b92-49ac-b912-64b943762b7e"><ac:rich-text-body>
<p>Depending on you PostgreSQL configuration you may need to add passwords to this file. If you do then be aware that, by default, this file is readable by all users. You should restrict access to just the hedgehog user on a Data Manager only machine or the hedgehog and www-data user on a combined Data Manager and Web GUI.</p></ac:rich-text-body></ac:structured-macro>
<p><br /></p>
<h2>Database configuration</h2>
<h3>Create a database</h3>
<p>Ask your DBA to create the necessary database. This is a script to help them. It creates the DATABASE, SCHEMA, FUNCTIONS, EXTENSIONS, LANGUAGES, USERS and ROLES needed to run hedgehog (using default values), and can optionally accept a user specified database name and read/write user names.</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="16c3ab93-2380-4a94-8c94-8a9a71249823"><ac:plain-text-body><![CDATA[sudo -u postgres /usr/bin/hedgehog_database_create
 
#If you want to use passwords do something like this or use peer authentication
sudo -u postgres psql
alter user hedgehog password 'hedgehog123';
alter user "www-data" password 'www123';]]></ac:plain-text-body></ac:structured-macro>
<p>Modify the PostgreSQL configuration as so:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="e4620eed-e360-454c-9590-b130d1ca55e8"><ac:plain-text-body><![CDATA[sudo vi /etc/postgresql/9.3/main/postgresql.conf
# uncomment and set 'extra_float_digits = 1'
sudo service postgresql reload]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="8916bafd-dbfe-4999-ba7c-b5a4a4417c2b"><ac:rich-text-body>
<p>We recommend that the 'pgtune' tool is used to obtain an optimal configuration for PostgreSQL. For example:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="48a4dfb3-f0c2-4f13-9942-c1a7c69069d8"><ac:plain-text-body><![CDATA[pgtune -i /etc/postgresql/9.3/main/postgresql.conf -c 200 -T DW]]></ac:plain-text-body></ac:structured-macro>
<p>Also some queries to the database trigger DNS lookups from functions in the database. It may be optimal to run a local resolver such as Unbound.</p></ac:rich-text-body></ac:structured-macro>
<p><br /></p>
<h2><span>Data Manager configuration</span></h2>
<h3>Specify the nodes/servers</h3><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="33b7d9dd-5c22-40dd-b1cb-2cc833ff1b7a"><ac:rich-text-body>
<p><span>For this version of Hedgehog the servers and nodes to be processed and displayed must be specified manually as described here.</span></p></ac:rich-text-body></ac:structured-macro>
<ul>
<li>Edit the the&nbsp;<em>/etc/hedgehog/nodes.csv</em>&nbsp;file to specify the servers, nodes and grouping to be used (example format is provided with entries commented out).</li>
<li>Note that the current GUI layout is optimised for nodes with short names (&lt;6 characters) of the same length</li></ul>
<h3>Specify the server service addresses (if needed)</h3>
<p><span>If you want to restrict the server IP addresses that are displayed in the 'By server IP address table' then edit the <em>/etc/hedgehog/service_addr.csv</em> file to add the list of addresses to be displayed.</span></p>
<h3><span>Directory permissions</span></h3>
<p><span>The </span>/var/lib/hedgehog/data/ is used to store incoming XML files</p><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="c988fd5b-f033-4348-a2e6-38dcd2795b97"><ac:rich-text-body>
<p>You may also need to alter the permissions on this directory to allow uploads via your chosen mechanism</p></ac:rich-text-body></ac:structured-macro>
<h3>Create the database tables</h3>
<p>Run the command below noting the following:</p>
<ul>
<li>If you have historic data to import then use the -m flag to specify the month of the oldest data that will need importing. Otherwise the database tables will be created to hold data from this month onwards.</li>
<li>Note that this script will also create the directory structure for all the specified servers and nodes under the <em>data</em> directory if it does not exist</li>
<li>(Note the insertion of the GeoIP data can take some time)</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="fe8954a6-8071-4c6c-917f-f2af4cfc7e64"><ac:plain-text-body><![CDATA[sudo -u hedgehog /usr/bin/hedgehogctl database_init]]></ac:plain-text-body></ac:structured-macro>
<h3>(Optional) Webdav upload</h3>
<p>If using webdav to upload XML files then add the following to the /etc/apache2/envvars file:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="38cd9a9c-70a2-4865-9c72-811dda53e3d2"><ac:plain-text-body><![CDATA[umask 002]]></ac:plain-text-body></ac:structured-macro>
<p>Then enable the dependancies:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="3bcca21b-ddef-46de-9f3e-d4f4d4d5edbd"><ac:plain-text-body><![CDATA[sudo a2enmod dav
sudo a2enmod ssl
suod a2enmod dav_fs]]></ac:plain-text-body></ac:structured-macro>
<p>And then enable the hedgehog webdav site:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="5136fdd1-28a2-4901-a344-a7702383709d"><ac:plain-text-body><![CDATA[sudo a2ensite hedgehog-webdav-upload]]></ac:plain-text-body></ac:structured-macro>
<p><br /></p>
<h2>Web GUI configuration</h2>
<p>Check the parameters in the/etc/hedgehog/hedgehog_gui.yaml file, which specifies parameters controlling the behaviour of the web front end. See the &quot;Plot Caching&quot; section in the user guide for a more detailed description of when plots are cached.</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="dd531aa2-f9a3-4f1d-a917-9ca914ec4530"><ac:plain-text-body><![CDATA[--- 
# YAML config for hedgehog GUI.
# NOTE: If this file is changed then apache must be restarted for the changes to take effect
www:
  default_plot_type             : interactive # 'static'      -> png plots
                                              # 'interactive' -> googlevis plots
  default_interactive_plot_type : svg         # 'flash' -> plot requires flash
                                              # 'svg'   -> plot is SVG/VML and 
                                              # does not require flash (but with svg
                                              # plots some legends do not wrap properly)
  default_node_grouping         : instance    # choose from 'none', 'instance', 'city' or
                                              # 'country'
  use_plot_caching              : 1           # '1' -> true, use cached plots when possible
                                              # '0' -> false, never use cached plots
  caching_delay_in_hours        : 1           # If 'use_plot_caching=1' then only plots with
                                              # an end time earlier than this number of 
                                              # hours ago are cached. More recent plots are
                                              # not cached as data may still be importing
  presentation_delay_in_hours   : 0           # Number of hours behind now for which the
                                              # GUI will display data
  support_url                   :             # configurable target for "Support" external
                                              # link on Homepage. The default dns-stats.org
                                              # issue tracker used if left blank
  default_server                :             # Optionally specify the default server to
                                              # use in the server drop-down 
                                              # (default is first alphabetically)
  default_plot_name             : by_region   # default plot name (See Hedgehog internal 
                                                plot name in the user guide)
  enable_node_selection         : 1           # '1' -> true, Show the node radio button
                                              # '0' -> false, hide the node radio button]]></ac:plain-text-body></ac:structured-macro>
<h3>Apache configuration</h3>
<p>Depending on your exact installation choices and apache configuration y<span style="line-height: 1.4285715;">ou may need to disable the default site using the following command:</span></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="98d7d7c3-f17d-4110-985b-113256902536"><ac:plain-text-body><![CDATA[sudo a2dissite 000-default.conf]]></ac:plain-text-body></ac:structured-macro>
<p><br /></p>
<ul>
<li>Add the Hedgehog configuration files to apache and enable the site (this file name can be changed if required to match any local apache policy):</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="01f53792-d09e-4d76-886f-7cf5b123ac37"><ac:plain-text-body><![CDATA[sudo a2ensite hedgehog.conf]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="be50c57a-e595-474b-a476-1449dc6766a8"><ac:rich-text-body>
<p><span>apache/rapache write some of their logs to user.* so it can be useful to change the syslog config: </span></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="d573ab6b-d29e-44e4-8d87-41fe0a8c74ea"><ac:plain-text-body><![CDATA[sudo vi /etc/rsyslog.d/50-default.conf]]></ac:plain-text-body></ac:structured-macro>
<p>Uncomment the line&nbsp;beginning&nbsp;'user.*'.&nbsp;</p></ac:rich-text-body></ac:structured-macro>
<p><br /></p>
<ul>
<li>Finally, reload apache:</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="744ab3cb-e4f9-43f0-8174-db3f3eda513e"><ac:plain-text-body><![CDATA[sudo service apache2 reload]]></ac:plain-text-body></ac:structured-macro>
<p><br /></p>
<p>At this point you should test that you can see the servers and nodes in the web front end at the URL http://&lt;server-name&gt;/hedgehog</p>
<h1>Importing data</h1>
<p>Hedgehog can process data in the following 3 ways:</p>
<table class="wrapped">
<tbody>
<tr>
<th>Source format</th>
<th>Output format</th>
<th><br /></th></tr>
<tr>
<td>XML</td>
<td>Database</td>
<td>For real time uploads</td></tr>
<tr>
<td colspan="1">DAT</td>
<td colspan="1">Database</td>
<td colspan="1">For import of historic data</td></tr>
<tr>
<td colspan="1">XML</td>
<td colspan="1">DAT</td>
<td colspan="1">For backwards compatibility with DSC</td></tr></tbody></table>
<p><br /></p>
<p>In each case the <em>/usr/bin/refile_and_grok</em> script is used, it is simply given different parameters:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="a0503194-2b02-4ad3-8826-e8c87d2c93d7"><ac:plain-text-body><![CDATA[> refile_and_grok -h

refile_and_grok - finds all input files in working directory and processes to output format


-w Working directory to search for input files (default: )
-i Input file format <XML|DAT> (default: XML)
-o Output file format <DAT|DB> (default: DB)
-c Non-interactive mode - use this flag when being run by a cron job
-s Start date from which to process incoming data (XML input only)
-r Disable processing of rssac data. Default is to process all data.
-R Reserved processors. Number of CPUS processors to exclude from import (default 0).
-a Append output to the refile_and_grok.stdout file (default is overwrite)
-h Show this help.]]></ac:plain-text-body></ac:structured-macro>
<p><br /></p>
<h2>Importing historical .DAT data</h2><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="8937cd73-d9f5-4af6-b41d-56e9ef9b925c"><ac:plain-text-body><![CDATA[sudo -u <DB_OWNER> /usr/bin/refile_and_grok -i DAT]]></ac:plain-text-body></ac:structured-macro>
<p>Be aware that this can take a long time if there is a significant amount of historic data and it may be advisable to run this in stages.</p>
<h2>Importing real-time .XML data</h2>
<h3>Manually</h3>
<ul>
<li>This can be done manually by running the <em>refile_and_grok</em> script (consider running this nohup as it may take a while depending on how much data there is to process).</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="71957016-03d6-4d53-98b6-847f9b2e0ddb"><ac:plain-text-body><![CDATA[sudo -u <DB_OWNER> /usr/bin/refile_and_grok]]></ac:plain-text-body></ac:structured-macro>
<ul>
<li>A snapshot of the progress of the data import can be generated by running the command below:</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="51ca4ae5-494c-47f2-aacd-ca30d38c6019"><ac:plain-text-body><![CDATA[sudo -u <DB_OWNER> /usr/bin/hedgehogctl datafiles_create_summary]]></ac:plain-text-body></ac:structured-macro>
<h3>Automatically</h3>
<ul>
<li>Configure a regular cron job for <em>refile_and_grok</em> as shown below</li></ul>
<h2>Importing zone-size and load-time data for RSSAC</h2>
<p>To do this run&nbsp;the&nbsp;<em>rssacd</em>&nbsp;demon specifying the server of interest. This listens for NOTIFY messages and after receiving one:</p>
<ul>
<li>obtains the zone size from the notifying server by performing an XFR and&nbsp;</li>
<li>probes each node listed for the configured server&nbsp;with an IP address&nbsp;to calculate the zone load time&nbsp;</li></ul><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="ff2b3e0a-8fe8-406f-8c30-b0b73bbf70d1"><ac:rich-text-body>
<p><span>This requires that management IP addresses are configured for the nodes via the nodes.csv file (also note that&nbsp;</span><em>rssacd</em><span>&nbsp;needs restarting if new nodes are added).</span></p></ac:rich-text-body></ac:structured-macro>
<p><em style="line-height: 1.42857;">rssacd</em>&nbsp;can be run from the command line for testing, or using an init script.</p>
<p>From the command line:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="c5b57e0c-4417-415f-9f13-35848a6a5ec8"><ac:plain-text-body><![CDATA[/usr/sbin/rssacd --log ~/var/log/hedgehog/rssacd.log -s <server-name> -z <fully_qualified_zone_name>]]></ac:plain-text-body></ac:structured-macro>
<p>From an init script:</p>
<ul>
<li>Configure the server name, TSIG key, etc. in the /etc/hedgehog/rssac.conf file</li></ul>
<p>Then run:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="fad2282a-24f0-44d9-98bf-e718ff93ea98"><ac:plain-text-body><![CDATA[update-rc.d rssacd defaults]]></ac:plain-text-body></ac:structured-macro>
<p><br /></p>
<h1>Cron jobs</h1>
<p>Several cron jobs need to be configured.</p>
<h2>Data Manager component</h2>
<p>Below is an example crontab for a typical data manager install (<em>sudo -u &lt;DB_OWNER&gt; crontab -e</em>).</p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="f24fdc6c-30e0-4911-ae3b-aef0fdccb1e9"><ac:rich-text-body>
<p><strong><span>Note that the </span><em>database_manage_partitions </em><span>script </span><strong>MUST </strong> be run at least once a month to create the tables for next month or the import will fail.</strong></p></ac:rich-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="b841f66d-ab0d-49cf-8c1d-0d7178f85e3d"><ac:plain-text-body><![CDATA[# REQUIRED: 
# Import XML data every 15 mins
00,15,30,45 * * * * /usr/bin/refile_and_grok -c >> /var/log/hedgehog/refile_and_grok_xml_to_db.log 2>&1
# Twice monthly job to make sure the DB tables for next month are created 
# ahead of time
0 6 15,28 * * /usr/bin/hedgehogctl database_manage_partitions >> /var/log/hedgehog/database_manage_partitions.log 2>&1

# OPTIONAL:
# Daily jobs to process RSSAC data. By default data is processed 
# for a single day 1 week ago. Must be run before the rssac_generate_reports script (see Web GUI Component cron jobs)
0 1 * * * /usr/bin/hedgehogctl database_process_rssac_data -D >> /var/log/hedgehog/database_process_rssac_data.log 2>&1
# Weekly job to update the delegated TLDs from the IANA database.
0 2 * * 0 <prefix>/bin/hedgehogctl database_update_tlds_from_zone >> /var/log/hedgehog/database_update_tlds_from_zone.log 2>&1
# Monthly job to update the geoIP database.
0 3 1 * * <prefix>/bin/hedgehogctl database_update_geoip >> /var/log/hedgehog/database_update_geoip.log 2>&1
# Monthly job to remove empty xml directories that are older than 7 days old
0 2 1 * * /usr/bin/hedgehogctl datafiles_rm_empty_xml_dirs -D >> /var/log/hedgehog/datafiles_rm_empty_xml_dirs.log 2>&1
# Monthly job to tar up processed xml directories
0 2 7 * * /usr/bin/hedgehogctl datafiles_tar_old_xml -D >> /var/log/hedgehog/datafiles_tar_old_xml.log 2>&1]]></ac:plain-text-body></ac:structured-macro>
<h2>Web GUI component</h2>
<p>Below is an example crontab for a typical web front-end install (<em>sudo -u &lt;DB_READ_USER&gt; crontab -e</em>)</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="760d5b53-0421-4e33-87d2-61beff00d451"><ac:plain-text-body><![CDATA[# OPTIONAL: 
# Daily job to create cached plots for the previous day to make loading common plots  
# quicker. Run a few hours after midnight so all data is uploaded.
0 4 * * * /usr/bin/hedgehogctl plotcache_generate_cached_plots -D >> /var/log/hedgehog/plotcache_generate_cached_plots.log -D 2>&1
# Daily job to generate RSSAC reports. By default report is generated 
# for a single day 1 week ago. Must be run after the database_process_rssac_data script (see Data Manager cron jobs)
0 3 * * * /usr/bin/hedgehogctl rssac_generate_reports >> /var/log/hedgehog/rssac_generate_reports.log 2>&1]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p></ac:rich-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p>

<p>The Hedgehog tools are a set of scripts that are used to perform various tasks in terms of creating and maintaining the database and hedgehog files on disk. They can be run directly from <span>/libexec/hedgehog or</span>&nbsp;for convenience a wrapper script (<span>/bin/hedgehogctl</span>) is provided:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="b3c2535a-74e2-4382-956a-66431f19a058"><ac:plain-text-body><![CDATA[>hedgehogctl help
 
Run a Hedgehog command

Usage: hedgehogctl [help|list] [COMMAND] [command_options]]]></ac:plain-text-body></ac:structured-macro>
<p>The available hedgehogctl commands are described below</p>
<p><ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="8fff6759-8a52-4a19-929b-bc3886a8c215" /></p>
<h2>Data Manager</h2>
<p>These scripts should be run as the DB_OWNER user (<em>hedgehog</em> by default).</p>
<h3>Database</h3>
<h4>database_init</h4>
<ul>
<li>The script should be run after the database has been created (see the &lt;prefix&gt;/libexec/database_create helper script)</li>
<li>This script populates the database with the plot and server/node data</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="35c6307d-3935-4e00-bffb-e37aa1ac19b6"><ac:plain-text-body><![CDATA[Create and populate the database tables with plot options and nodes.

This calls the hedeghog_database_update_nodes.sh script which will populate the DB
with the specified nodes and create a directory structure for them in the working
data directory.

Usage: database_init options

Supported options:

  -m Start month of oldest data to be imported 
     (default is current month, format YYYY-MM)
  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h4>database_manage_partitions</h4>
<ul>
<li>
<p>This script creates new tables in the database to hold the imported data.&nbsp;</p><ac:structured-macro ac:name="warning" ac:schema-version="1" ac:macro-id="02fd2867-ebc4-40f4-ac80-9a3515fc90a6"><ac:rich-text-body>
<p><strong>It must be run at least once a month to create the tables for next month or the import will fail. It is recommended to configure a cron job as described in the&nbsp;Cron jobs section of the Installation guide.&nbsp;</strong></p></ac:rich-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p></li></ul>
<ul>
<li>Otherwise, the user will only have to run this script when new servers are added to an existing system.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="d5d85645-ac8c-4f52-a28d-93142894bb06"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Create a new set of partitioned data tables in the database to hold imported data.
The partitions are created per month of data.
 
Usage: database_manage_partitions options

Supported options:
    -m Month to create partitions for 
       (default is next month, format YYYY-MM)
    -h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h4>database_process_rssac_data</h4>
<ul>
<li>This script should be run to process the data ready for the <em>rssac_process_data </em>script.&nbsp;</li>
<li>The importer inserts data from the XML files into the 'unique_source_raw' table. For each node, this script creates entries in the 'unique_source_summary' table based on the raw data and then (optionally) deletes the original raw data from the&nbsp;'unique_source_raw' table.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="1c107461-c555-46a2-aed9-0eb3d0823078"><ac:plain-text-body><![CDATA[Process RSSAC data for a specified 24 hour period
 
Usage: database_process_rssac_data options

Supported options:
  -d Date on which to process RSSAC data 
    (default is 7 days ago, format: YYYY-MM-DD)
  -f Force overwrite of existing summary data
  -D Also delete the raw unique_source data from the database to save space 
    (default: no delete)
  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h4>database_rm_old_rssac_data</h4>
<ul>
<li>For some systems, the unique source raw data can be unmanageably large. This script can be used to truncate the database tables to recover the disk space, for example it can be run once a month to remove all the raw data for the previous month</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="af1ff305-093e-450e-88c7-fd6c5cbc7f1b"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Delete all the raw unique_source data in the database for a given month. 
All data older than this is also deleted.
This script is intended to be run monthly to free disk space on systems 
where the raw unique sources data becomes unmanagably large.

Usage: database_rm_old_rssac_data options

Supported options:
  -m Month from which to delete all raw unique source data 
       (default 2014-07: format YYYY-MM)
       (default is 2 months ago when in the first half of the current month and
        1 month ago when in the second half of the current month)
  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h4>database_update_geoip</h4>
<ul>
<li>This script downloads a reference database of locations for use by the GEO maps. It should be periodically run to update with new location information.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="660d7d6b-270c-41fa-8cba-cd4f95aeb632"><ac:plain-text-body><![CDATA[Update the GeopIP list in the database. This reads input from /tmp
Usage: database_update_geoip options
Supported options:
  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
<h4>database_update_nodes</h4>
<ul>
<li>Run this script if new nodes or servers are added to the system</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4d9265e8-3cd1-4f78-b752-2861b5671147"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Add/update the servers and nodes in the database. This reads input from file
called nodes.csv in the /usr/local/etc/hedgehog directory.
 - An example nodes.csv file is installed if one does not exist.
 - See the comments in that file for details of the format.
 - No action is taken on servers/nodes that are in the database but are not in the 
   input files.

Usage: /usr/local/bin/hedgehog_database_update_nodes options

Supported options:
  -r rename/remove flag     This enables processing of the rename_or_remove field in nodes.csv
                            If this flag is not given any lines in the input file that have
                            text in the rename_or_remove field are ignored.
                            Renaming a node will update the database and move the data
                            directory to the new name.
                            Removing a node will delete all associated data in the database
                            but leave any data on disk untouched (for manual removal).
                            Removing nodes could take a long time if you have a lot of data.
  -d debug mode (set -x).
  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
<h4>database_update_service_addr</h4>
<ul>
<li>Run this script to control which server addresses are displayed in the 'By server IP address' plot. This can be useful to filter to just the server service addresses since DSC does not have such a fitler.&nbsp;</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="e88fc06e-c141-4890-89cb-321223aa31d4"><ac:plain-text-body><![CDATA[Update the Server service addreses list in the database.

This list is only used to filter which addresses are displayed in the
'By server IP address' plot. If there are any addresses in the list
then only these addresses are displayed. If there are no entries 
in the list then all the addresses for which there is data are displayed.
By default this script reads the /opt/hedgehog/etc/hedgehog/service_addr.csv file and adds
the addresses found there.
After running this script it is advisable to remove any cached plots
for the 'By server IP address' plot by using the plotcache_rm_cached_plots
script.

Usage: database_update_service_addr options

Supported options:
  -d Delete all existing entries in the server service address list
  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
<p><br /></p>
<h4>database_update_tlds_from_zone</h4>
<ul>
<li>Run this script to update the categorisation of TLDs in the database from the root zone held in the IANA database.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="7ef95ada-9bd4-46b3-b66a-ba2cfbd01256"><ac:plain-text-body><![CDATA[Update the TLD list in the database. This reads input from IANA ftp site
from the directory.
No action is taken on TLDs that are in the database but are not in the input data.
Usage: database_update_tlds_from_zone options
Supported options:
  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
<h3>Datafiles</h3>
<h4>datafiles_create_summary</h4>
<ul>
<li>This is a useful utility script that can be used to get a high level picture of the processing of the XML files for all the nodes in the system. It can help identify backlogs in the processing system.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="bdc7b96f-0f56-4d9d-a73b-ce13df2d9d2b"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Generate a processing report based on the information in the log files

Usage: datafiles_create_summary options

Supported options:
	-s Only report on the server with this name (default is all servers)
	-n Only report on the node with this name (default is all nodes)
	-c Output in csv format for import into spreadsheet application
    -d Run in debug mode (set -x)
	-h Show this help.]]></ac:plain-text-body></ac:structured-macro>
<h4>datafiles_rm_empty_xml_dirs</h4><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="88bd5715-ba68-4a20-a7f1-be499d770540"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Remove empty incoming xml directories after a certain period of time 
(default is all the processed directories older than 7 days ago)

Usage: datafiles_rm_empty_xml_dirs options

Supported options:
	-d Date before which removing empty xml directories 
      (default: 7 days ago, format YYYY-MM-DD)
	-h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h4>datafiles_tar_old_xml_files</h4><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="9283cb80-b944-415c-8f1b-065ade9f307f"><ac:plain-text-body><![CDATA[Pack old xml files already processed into a .tar package to store and archive.
The xml files are packed according to their date and node and server.
The filename has the following format done-PID-date.tar.bz2 (e.g. done-14143-2015-05-01.tar.bz2).

Usage: datafiles_tar_old_xml_files options

Supported options:
  -h Show this help.]]></ac:plain-text-body></ac:structured-macro>
<h2>Web front-end</h2>
<p>These scripts should be run as the DB_READ_USER user (<em>www-data</em> by default).</p>
<h3>Plotcache</h3>
<h4>plotcache_generate_cached_plots</h4>
<ul>
<li>
<p>You must add at least one option for which plot window to generate [-D|-W|-M].</p></li>
<li>
<p>By default, the cached plots are generated daily for the day before between 00:00 and 23:59. It can be helpful to configure a daily cron job to use this script to create cached plots to make loading of the homepage and common plots faster.&nbsp;</p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="e3a20735-7555-41f6-8f0f-1b885b193085"><ac:rich-text-body>
<p><span>If not relying on the default behaviour be careful to specify BOTH a start and end date for this script, otherwise the end date will default to today and this may result in many plots being generated.</span></p></ac:rich-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p></li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4a423b5a-ca47-4597-99b9-9e7da0d06d90"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Create cached plots for each chosen period (day/week/month) between Start and End range of time.

Usage: generate_cached_plots options

Supported options:
    -s Start date of time range from which to create cached plots 
       (default is 1 day before at 00:00, format YYYY-MM-DDThh:mm)
    -e End date of time range from which to create cached plots 
       (default is 1 day before at 23:59, format YYYY-MM-DDThh:mm)
    -D Generate daily cached plots
    -W Generate weekly cached plots
    -M Generate monthly cached plots
    -h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h4>plotcache_rm_cached_plots</h4>
<ul>
<li>
<p>The user should not normally need to run this script, but is available in case problem are encountered with cached plots, or it is desirable to flush the cache for some reason (for example the default interactive plot type is changed after graphs have been cached).&nbsp;</p></li>
<li>
<p>This script is interactive if the -a option is not selected.&nbsp;You will be asked to select which cached plot types you'd like to remove.</p></li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="839813d2-ab5b-4555-a419-880edcf01815"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Remove all or a set of cached plots from the directory structure.
 
Usage: plotcache_rm_cached_plots options

Supported options:
  -a Remove all cached plots
  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h3>RSSAC</h3>
<h4>rssac_generate_reports</h4>
<ul>
<li>By default the script creates RSSAC reports for all configured servers for a date 7 days in the past</li>
<li>It will require that the&nbsp;database_process_rssac_data script will have already been run for the date in question.<em>&nbsp;</em></li>
<li>Note the unique_source data for the report is taken from the summary table, not the raw table.</li></ul><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="31543074-e163-4040-8ccd-a8ec5c66780c"><ac:parameter ac:name="language">text</ac:parameter><ac:plain-text-body><![CDATA[Create RSSAC reports for a specified 24 hour period
 
Usage: rssac_generate_reports options

Supported options:
  -d Date on which to create RSSAC reports 
     (default is 7 days ago, format: YYYY-MM-DD)
  -h Show this help]]></ac:plain-text-body></ac:structured-macro>
<h2>Other tools</h2>
<h3>conf_read</h3>
<ul>
<li>The script parses the contents of the <em>&lt;prefix&gt;/etc/hedgehog/hedgehog.yaml</em> file into environment variables for use my the other scripts. Users will not need to run this directly.</li>
<li>The database parameters are to be set by the user (if default ones need to be modified) and the directory structure parameters are auto-generated and better left alone.</li></ul>

<p><br /></p>
<p>This document describes how to upgrade a server from release 2.4.1 to release 2.4.2 of Hedgehog via packages.&nbsp;</p>
<h2>Release 2.4.2</h2>
<p>There are no Hedgehog actions required for this upgrade.</p>
<p><br /></p>

<p><ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="a93a02e9-93ad-4b13-b26a-5c1cb6b996ce" /></p>
<h1>Overview of Hedgehog</h1>
<p>Hedgehog is a visualisation tool for DNS statistics that consumes data&nbsp;acquired with the DSC collector.</p>
<p>Hedgehog was initially developed for ICANN by Sinodun IT and is now released&nbsp;open source under the MPL License. For more information on the development and support of Hedgehog see the <a href="http://www.dns-stats.org">http://www.dns-stats.org</a>&nbsp;website.</p>
<h1>Components</h1>
<p>Hedgehog comprises 3 components which can be run on the same or different machines:</p>
<ul>
<li>A Postgres Database</li>
<li>Data Manager
<ul>
<li>Scripts for database creation and management</li>
<li>XML/DAT processing</li></ul></li>
<li>Web GUI front-end</li></ul>
<h2>Installation options</h2>
<p>Hedgehog can be installed to run on a single server with all the components local to that server.&nbsp;</p>
<p><ac:image ac:height="400"><ri:attachment ri:filename="Hedgehog_component_view.jpg" /></ac:image></p>
<p>Alternatively the DB may be on a remote server and the Data Manager and Web front-end may be on different servers as shown below.</p>
<p><ac:image ac:height="400"><ri:attachment ri:filename="Hedgehog_component_view_split.jpg" /></ac:image></p>
<h3>Install from Packages (Ubuntu)</h3>
<p>From 2.2 packages for Ubuntu are provided for Hedgehog. It is recommended to use this method for new installs.</p>
<h3>Install from Source Code</h3>
<p>Hedgehog can also be installed directly from source code if desired - note: this is not a supported method.</p>
<h1>Datasets</h1><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="1dccef1c-27f8-4b28-84b4-ca7010c69dc6"><ac:rich-text-body>
<p>Hedgehog is currently tested against the output of the original DSC collector. It has not been tested against the recently released DSC 2.0.</p></ac:rich-text-body></ac:structured-macro>
<p>&nbsp;</p>
<p>An example dsc.conf file for the data sets that Hedgehog supports is given below:</p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4b93a047-e94d-4b6b-a945-fb6e5ec2414a"><ac:plain-text-body><![CDATA[# Basic data sets
dataset qtype dns All:null Qtype:qtype queries-only;
dataset rcode dns All:null Rcode:rcode replies-only;
dataset opcode dns All:null Opcode:opcode queries-only;
dataset rcode_vs_replylen dns Rcode:rcode ReplyLen:msglen replies-only;
dataset client_subnet dns All:null ClientSubnet:cip4_net queries-only max-cells=200;
dataset qtype_vs_qnamelen dns Qtype:qtype QnameLen:qnamelen queries-only;
dataset qtype_vs_tld dns Qtype:qtype TLD:tld queries-only,popular-qtypes max-cells=200;
dataset certain_qnames_vs_qtype dns CertainQnames:certain_qnames Qtype:qtype queries-only;
dataset client_subnet2 dns Class:query_classification ClientSubnet:cip4_net queries-only max-cells=200;
dataset client_addr_vs_rcode dns Rcode:rcode ClientAddr:client replies-only max-cells=50;
dataset chaos_types_and_names dns Qtype:qtype Qname:qname chaos-class,queries-only;
dataset idn_qname dns All:null IDNQname:idn_qname queries-only;
dataset edns_version dns All:null EDNSVersion:edns_version queries-only;
dataset do_bit dns All:null D0:do_bit queries-only;
dataset rd_bit dns All:null RD:rd_bit queries-only;
dataset ipv6_rsn_abusers dns All:null ClientAddr:client queries-only,aaaa-or-a6-only,root-servers-net-only max-cells=50;
dataset transport_vs_qtype dns Transport:transport Qtype:qtype queries-only;
dataset direction_vs_ipproto ip Direction:ip_direction IPProto:ip_proto any;
dataset dns_ip_version_vs_qtype dns IPVersion:dns_ip_version Qtype:qtype queries-only;]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="eb6a3061-2017-49ee-aad8-6b96ad729e08"><ac:plain-text-body><![CDATA[# Additional data sets for RSSAC
dataset unique_sources dns IPVersion:dns_ip_version ClientAddr:client queries-only; 
dataset traffic_volume_queries dns Transport:transport IPVersion:dns_ip_version queries-only; 
dataset traffic_volume_responses dns Transport:transport IPVersion:dns_ip_version replies-only; 
dataset traffic_sizes_queries dns Transport:transport MsgLen:msglen queries-only; 
dataset traffic_sizes_responses dns Transport:transport MsgLen:msglen replies-only; ]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="4eaba3e3-a548-4198-89ac-4dfc069be91f"><ac:plain-text-body><![CDATA[# New in 2.1
dataset server_addr dns All:null ServerAddr:server queries-only;
# New in 2.2.0
dataset qtype_vs_client_subnet dns Qtype:qtype ClientSubnet:cip4_net queries-only max-cells=200;]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="code" ac:schema-version="1" ac:macro-id="7fd11eb1-b417-4b2a-8095-555c12a64e33"><ac:plain-text-body><![CDATA[# New in 2.4.0
dataset server_addr_vs_trans dns Transport:transport ServerAddr:server queries-only;]]></ac:plain-text-body></ac:structured-macro>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="26a887fa-d2f8-4988-9415-ab76c5e22739"><ac:rich-text-body>
<p>Data collection for the server_addr dataset is not yet supported in the official release of version 1 of DSC, but is available using the latest code from the DSC github repo. Also, note this dataset will contain the source addresses of all responses (QR = 1) seen on the collection node even if they do not originate from the local_address specified in the dsc.conf (i.e. if another server sends a response to the collection node for some reason).</p></ac:rich-text-body></ac:structured-macro>
<p>&nbsp;</p>
<h1>Plots</h1>
<p>The relationship between the Hedgehog plots and those produced by the DSC Presenter are listed here for convenience.</p>
<table><colgroup><col /><col /><col /><col /></colgroup>
<tbody>
<tr>
<th>Hedgehog plot category</th>
<th colspan="1">Hedgehog plot display name</th>
<th>DSC (and Hedgehog internal) plot id</th>
<th>DSC display name</th></tr>
<tr>
<td colspan="1"><strong>Node Statistics</strong></td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">By node</td>
<td colspan="1">by_node</td>
<td colspan="1">By node</td></tr>
<tr>
<td>
<p class="p2"><span class="s1"><strong>Query Attributes</strong></span></p></td>
<td colspan="1">
<p class="p1">&nbsp;</p></td>
<td>
<p class="p1">&nbsp;</p></td>
<td>&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">
<p class="p2"><span class="s1">CHAOS queries</span></p></td>
<td colspan="1">
<p class="p2"><span class="s1">chaos_types_and_names</span></p></td>
<td colspan="1">CHAOS</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">DO bit</span></p></td>
<td>
<p class="p2"><span class="s1">do_bit</span></p></td>
<td>Query Attributes &gt; DO bit</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">EDNS version</span></p></td>
<td>
<p class="p2"><span class="s1">edns_version</span></p></td>
<td>Query Attributes &gt; EDNS version</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">IDN qnames</span></p></td>
<td>
<p class="p2"><span class="s1">idn_qname</span></p></td>
<td>Query Attributes &gt; IDN Qnames</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">OPCODE</span></p></td>
<td>
<p class="p2"><span class="s1">opcode</span></p></td>
<td>Opcodes</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">
<p class="p2"><span class="s1">RD bit</span></p></td>
<td colspan="1">
<p class="p2"><span class="s1">rd_bit</span></p></td>
<td colspan="1">Query Attributes &gt; RD bit</td></tr>
<tr>
<td>
<p class="p2"><span class="s1"><strong>QTYPE</strong></span></p></td>
<td colspan="1">&nbsp;</td>
<td>
<p class="p1">&nbsp;</p></td>
<td>&nbsp;</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">DNS queries by QTYPE</span></p></td>
<td>
<p class="p2"><span class="s1">qtype</span></p></td>
<td>Qtypes</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">DNSSEC queries by QTYPE</span></p></td>
<td>
<p class="p2"><span class="s1">dnssec_qtype</span></p></td>
<td>DNSSEC Qtypes</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">
<p class="p2"><span class="s1">Popular query names by QTYPE</span></p></td>
<td colspan="1">
<p class="p2"><span class="s1">certain_qnames_vs_qtype</span></p></td>
<td colspan="1">Popular Names</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">
<p class="p2"><span class="s1">QTYPE for most popular TLDs</span></p></td>
<td colspan="1">
<p class="p2"><span class="s1">qtype_vs_tld</span></p></td>
<td colspan="1">TLDs</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">Query Name Lengths by QTYPE</span></p></td>
<td>
<p class="p2"><span class="s1">qtype_vs_qnamlen</span></p></td>
<td>Qname Lengths</td></tr>
<tr>
<td>
<p class="p2"><span class="s1"><strong>RCODE</strong></span></p></td>
<td colspan="1">
<p class="p1">&nbsp;</p></td>
<td>
<p class="p1">&nbsp;</p></td>
<td>&nbsp;</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">Replies by RCODE</span></p></td>
<td>
<p class="p2"><span class="s1">rcode</span></p></td>
<td>Rcodes</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">Reply lengths by RCODE </span></p></td>
<td>
<p class="p2"><span class="s1">rcode_vs_replylen</span></p></td>
<td>Reply Lengths</td></tr>
<tr>
<td>
<p class="p2"><span class="s1"><strong>IP Protocol</strong></span></p></td>
<td colspan="1">
<p class="p1">&nbsp;</p></td>
<td>
<p class="p1">&nbsp;</p></td>
<td>&nbsp;</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">IP version</span></p></td>
<td>
<p class="p2"><span class="s1">dns_ip_version</span></p></td>
<td>IP version</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">
<p class="p2"><span class="s1">Queries by IP version, QTYPE</span></p></td>
<td colspan="1">
<p class="p2"><span class="s1">dns_ip_version_vs_qtype</span></p></td>
<td colspan="1">IP version &gt; Query types</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td colspan="1">
<p class="p2"><span class="s1">Received packets by IP protocol</span></p></td>
<td>
<p class="p2"><span class="s1">direction_vs_ipproto</span></p></td>
<td>IP protocols</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p2"><span class="s1">Transports carrying DNS queries</span></p></td>
<td>
<p class="p2"><span class="s1">transport_vs_qtype</span></p></td>
<td>DNS transport</td></tr>
<tr>
<td colspan="1"><strong>PCAP statistics</strong></td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">PCAP statistics</td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td>
<p class="p2"><span class="s1"><strong>Client Subnet Statistics</strong></span></p></td>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p1">&nbsp;</p></td>
<td>&nbsp;</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p2"><span class="s1">Busiest client subnets</span></p></td>
<td>
<p class="p2"><span class="s1">client_subnet_accum</span></p></td>
<td>Client Geography</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p2"><span class="s1">RCODE by client subnet</span></p></td>
<td>
<p class="p2"><span class="s1">client_addr_vs_rcode_accum</span></p></td>
<td>Rcodes by Client Address</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p2"><span class="s1">Root abusers</span></p></td>
<td>
<p class="p2"><span class="s1">ipv6_rsn_abusers_accum</span></p></td>
<td>IPv6 root abusers</td></tr>
<tr>
<td>
<p class="p2"><span class="s1"><strong>Classification</strong></span></p></td>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p1">&nbsp;</p></td>
<td>&nbsp;</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p2"><span class="s1">Query classification by client subnet (accum)</span></p></td>
<td>
<p class="p2"><span class="s1">client_subnet2_accum</span></p></td>
<td>Classification</td></tr>
<tr>
<td>
<p class="p1">&nbsp;</p></td>
<td>
<p class="p2"><span class="s1">Query classification by client subnet (count)</span></p></td>
<td>
<p class="p2"><span class="s1">client_subnet2_count</span></p></td>
<td>Classification &gt; count</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">
<p class="p2"><span class="s1">Query classifications</span></p></td>
<td colspan="1">
<p class="p2"><span class="s1">client_subnet2_trace</span></p></td>
<td colspan="1">Classification &gt; trace</td></tr></tbody></table>
<p>In addition Hedgehog is capable of displaying plots based on datasets defined in the RSSAC 002 document (see later for more details):</p>
<table>
<tbody>
<tr>
<th>Hedgehog plot category</th>
<th>Hedgehog plot display name</th>
<th colspan="1"><span>Hedgehog internal plot id</span></th></tr>
<tr>
<td><strong>RSSAC</strong></td>
<td>&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1"><span>RCODE volume</span></td>
<td colspan="1">rcode_volume</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Traffic sizes</td>
<td colspan="1">traffic_sizes</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Traffic volume</td>
<td colspan="1">traffic_volume</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Traffic volume differences</td>
<td colspan="1">traffic_volume_differences</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Unique sources</td>
<td colspan="1">unique_sources</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Zone size</td>
<td colspan="1">zones_size</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Zone propagation time</td>
<td colspan="1">load_time</td></tr></tbody></table>
<p>The following plots are new in 2.1</p>
<table>
<tbody>
<tr>
<th>Hedgehog plot category</th>
<th>Hedgehog plot display name</th>
<th colspan="1"><span>Hedgehog internal plot id</span></th></tr>
<tr>
<td><strong>Node Statistics</strong></td>
<td>&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td>&nbsp;</td>
<td>
<p>By instance</p></td>
<td colspan="1">by_instance</td></tr>
<tr>
<td>&nbsp;</td>
<td>By server IP address</td>
<td colspan="1">server_addr</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">By city</td>
<td colspan="1">by_city</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">By country</td>
<td colspan="1">by_country</td></tr>
<tr>
<td colspan="1"><strong>QTYPE vs TLS</strong></td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">QTYPE for most popular Undelegated TLDs</td>
<td colspan="1">qtype_vs_othertld</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">QTYPE for most popular ccTLDs</td>
<td colspan="1"><span>qtype_vs_cctld</span></td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">QTYPE for most popular Legacy TLDs</td>
<td colspan="1"><span>qtype_vs_legacytld</span></td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">QTYPE for most popular New TLDs</td>
<td colspan="1"><span>qtype_vs_newtld</span></td></tr>
<tr>
<td colspan="1"><strong>Client Subnet Statistics</strong></td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Busiest client ASNs &nbsp;</td>
<td colspan="1">asn</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Busiest client BGP prefix</td>
<td colspan="1">bgpprefix</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">RCODE by client ASNs &nbsp;</td>
<td colspan="1">
<p class="p1"><span class="s1">client_addr_vs_rcode_accum_asn</span></p></td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">RCODE by clients by BGP prefix&nbsp;</td>
<td colspan="1">
<p class="p1"><span class="s1">client_addr_vs_rcode_accum_bgpprefix</span></p></td></tr>
<tr>
<td colspan="1"><strong>Classification</strong></td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Query classification by clients by BGP prefix</td>
<td colspan="1">
<p class="p1"><span class="s1">client_subnet2_accum_bgpprefix&nbsp;</span></p></td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Query classification by client ASNs</td>
<td colspan="1">
<p class="p1"><span class="s1">client_subnet2_accum_asn</span></p></td></tr>
<tr>
<td colspan="1"><strong>RSSAC</strong></td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">
<p>Zone size</p></td>
<td colspan="1">zone_size</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Zone propagation time</td>
<td colspan="1">load_time</td></tr>
<tr>
<td colspan="1"><strong>GEO</strong></td>
<td colspan="1">&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Busiest client locations by country&nbsp;</td>
<td colspan="1">geochart</td></tr>
<tr>
<td colspan="1">&nbsp;</td>
<td colspan="1">Busiest client locations by city</td>
<td colspan="1">geomap</td></tr></tbody></table>
<p>The following plots are new in 2.2</p>
<table>
<tbody>
<tr>
<th>Hedgehog plot category</th>
<th>Hedgehog plot display name</th>
<th colspan="1"><span>Hedgehog internal plot id</span></th></tr>
<tr>
<td><strong>Client Subnet Statistics</strong></td>
<td>&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td>&nbsp;</td>
<td>
<p>DNSKEY Queries by ASN</p></td>
<td colspan="1">
<p class="p1"><span class="s1">dnskey_vs_asn</span></p></td></tr></tbody></table>
<p>The following plots are new in 2.3</p>
<table>
<tbody>
<tr>
<th>Hedgehog plot category</th>
<th>Hedgehog plot display name</th>
<th colspan="1">Hedgehog internal plot id</th></tr>
<tr>
<td><strong>Node Statistics</strong></td>
<td>&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td>&nbsp;</td>
<td>
<p>By region</p></td>
<td colspan="1">
<p class="p1"><span class="s1">by_region</span></p>
<div><span class="s1"><br /></span></div></td></tr></tbody></table>
<p>The following plots are new in 2.4</p>
<table>
<tbody>
<tr>
<th>Hedgehog plot category</th>
<th>Hedgehog plot display name</th>
<th colspan="1">Hedgehog internal plot id</th></tr>
<tr>
<td><strong>Server IP</strong></td>
<td>&nbsp;</td>
<td colspan="1">&nbsp;</td></tr>
<tr>
<td>&nbsp;</td>
<td>
<p>By server IP address and transport</p></td>
<td colspan="1">
<p class="p1">server_addr_vs_trans</p></td></tr></tbody></table>
<p class="auto-cursor-target"><br /></p><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="f7a8bc07-2c7e-482e-850b-ad49fc4eae75"><ac:rich-text-body>
<ul>
<li><strong>RSSAC: Zone size and Zone propagation time graphs.</strong> In order for data to be available for these graphs, an instance of the <em>rssacd</em><span>&nbsp;demon&nbsp;must be run to collect the data (it is not provided by the DSC collector).</span></li>
<li><strong>GEO: Busiest client location by country.</strong> Both client locations and collection node locations can be displayed on this graph. For node locations to be displayed the nodes must have a city specified via the nodes.csv file.</li></ul></ac:rich-text-body></ac:structured-macro>
<h1>Directory Structure</h1>
<p>If installed from source code Hedgehog uses the following directory structure under the install &lt;prefix&gt; directory (/usr/local/ by default)</p>
<p><ac:image ac:width="550"><ri:attachment ri:filename="Hedgehog_directory_structure.jpg" /></ac:image></p>
<p>When installing from a Ubuntu package the structure is different:</p>
<table>
<thead>
<tr>
<th>Location (install from source)</th>
<th>
<div class="tablesorter-header-inner">Location (install from package)</div></th>
<th colspan="1">
<div class="tablesorter-header-inner">Contents</div></th></tr></thead>
<tbody>
<tr>
<td>&lt;prefix&gt;/etc/hedgehog/</td>
<td>/etc/hedgehog/</td>
<td colspan="1">Hedgehog configuration files (yaml and nodes)</td></tr>
<tr>
<td colspan="1">&lt;prefix&gt;/etc/hedgehog/*.sample</td>
<td colspan="1">/usr/share/doc/hedgehog/examples/*.sample</td>
<td colspan="1">Sample Hedgehog configuration files</td></tr>
<tr>
<td>&lt;prefix&gt;/var/hedgehog/</td>
<td>/var/lib/hedgehog/</td>
<td colspan="1">
<p>'data' directory containing XML data</p>
<p>'www' directory containing cached plots and rssac reports</p></td></tr>
<tr>
<td colspan="1">&lt;prefix&gt;/bin/hedgehogctl</td>
<td colspan="1">/usr/bin/hedgehogctl</td>
<td colspan="1">Wrapper script for Hedgehog tools</td></tr>
<tr>
<td colspan="1">&lt;prefix&gt;/bin/refile_and_grok</td>
<td colspan="1">/usr/bin/refile_and_grok</td>
<td colspan="1">refile_and_grok script</td></tr>
<tr>
<td colspan="1">&lt;prefix&gt;/libexec/hedgehog/</td>
<td colspan="1">/usr/lib/hedgehog/</td>
<td colspan="1">Hedgehog tools scripts and DDL updates</td></tr></tbody></table>
<p>In the rest of this document only the relative path under the install location is given so the instructions are applicable to both methods of install.</p>
<h1>Tools</h1>
<p>Hedgehog provides a set of Data Manager scripts to support the database and data file management, and Web scripts to generate cached plots and the RSSAC reports. They are documented separately in the Hedgehog Tools PDF.&nbsp;</p>
<h1>Data Manager</h1>
<h2>Upload of Data</h2>
<p>It is assumed that the user has already deployed an upload mechanism used with the DSC collector and presenter. A common practice is to use ssh for the upload.</p>
<p>Hedgehog also supports the use of WebDAV for uploads, validated with certificates. This is particularly useful for deployments with many nodes as it scales better than ssh. It can be configured with the hedgehog-webdav-upload package.</p>
<h2><span style="line-height: 1.25;">Import of data</span></h2>
<p>Hedgehog provides a refile_and_grok script for this purpose. See the Installation guide for more details of how to configure this.</p>
<h2>Monitoring data import</h2>
<p>The import process produces two different logs for the user to track the progress of the data import:</p>
<ul>
<li>The<em> refile_and_grok</em>&nbsp;script writes output about which nodes it is processing to a <em>refile_and_grok.out</em> file in the Hedgehog <em>/var/hedgehog/data</em> directory. This gives a high level view of the processing.&nbsp;<br />&nbsp;</li>
<li>The import also generates a log file per node in the <em>/var/hedgehog/data/&lt;server&gt;/&lt;node&gt;</em>&nbsp;directory called <em>dsc-extractor.out</em>. This lists more details on the processing of each XML file for a specific node.&nbsp;<br />&nbsp;</li>
<li>A utility script is also provided: <em>/bin/hedgehogctl datafiles_create_summary</em>&nbsp;which can be run to get an summary of the current state of processing.</li></ul>
<h2>Adding or updating servers and nodes</h2>
<p>If new servers or nodes need to be added to the system then the user should:</p>
<ul>
<li>update the <em>/etc/hedgehog/nodes.csv</em> files as required to include the new servers/nodes<br />&nbsp;</li>
<li>run the&nbsp;<em>/bin/hedgehogctl database_update_nodes</em>&nbsp;script which will update the data in the database and create any required directory structures</li></ul><ac:structured-macro ac:name="note" ac:schema-version="1" ac:macro-id="049c0d31-3cdd-4ef8-ac3c-ef5ad4e0e487"><ac:rich-text-body>
<p>In the event of adding new servers the user&nbsp;<strong>must</strong>&nbsp;also then run the&nbsp;<em>/bin/hedgehogctl database_manage_partition</em> scripts to create the required database tables so that data can be imported for the new server.&nbsp;</p></ac:rich-text-body></ac:structured-macro>
<h1>Web front-end</h1>
<h2>Plot caching</h2>
<p>Hedgehog caches plot image files as they are generated by the user. The plot files are cached in the directory listed in the <em>/etc/hedgehog/hedgehog.yaml</em> file in the directories section for 'plotcache' (this path is auto-generated in 2.0 and should not be changed by the user).</p>
<ul>
<li>The reuse of cached plots can be enabled/disabled by changing the <em>use_plot_caching</em>&nbsp;option in the /<em>share/www/conf/hedgehog_gui.yaml</em> configuration file<br />&nbsp;</li>
<li>The <em>caching_delay_in_hours</em>&nbsp;option in the same file controls the delay applied to caching the most recently uploaded data. For example the default for this is 1 i.e. only plots for which the end time is more than one hour ago will be cached. This avoids caching plots where the data may still be being imported for some nodes if there were a delay in upload or processing for some reason.&nbsp;<br />&nbsp;</li>
<li>If desired, a cron job can be created that will create cached plots for specific time periods, for example it may be useful to have a cron job to create daily plots for the previous day (see the section on Maintenance). This will make loading of the homepage faster.&nbsp;<br />&nbsp;</li>
<li>If needed the cached files for selected or all plots may be deleted by running the<em>&nbsp;/bin/hedgehogctl plotcache_rm_cached_plots</em>&nbsp;script.&nbsp;</li></ul>
<h2>RSSAC reports</h2>
<h3>Data collection</h3>
<p>If the required datasets are collected via the DSC collector (see Datasets above), Hedgehog can produce RRSAC reports as described in the &quot;<span style="line-height: 1.4285715;">&nbsp;</span><a href="https://www.icann.org/en/system/files/files/rssac-002-measurements-root-20nov14-en.pdf"><span style="line-height: 1.4285715;">RSSAC Recommendation on Measurements of the Root Server System -&nbsp;</span><span style="line-height: 1.4285715;">RSSAC 002</span></a><span style="line-height: 1.4285715;">&nbsp;</span>&quot; document for the following metrics:</p>
<ul>
<li>traffic-volume</li>
<li>traffic-size</li>
<li>rcode-volume</li>
<li>unique-sources</li>
<li>zone-size</li>
<li>load-time</li></ul>
<h3>RSSACD</h3>
<p>In order to collect the zone-size and load-time statistics a dedicated <em>rssacd</em> demon must be run on the Hedgehog Data Manager to collect the information directly from the nodes. It requires that the IP addresses of the nodes have been populated via the nodes.csv file. More details are given in the Installation/Upgrade Guides.</p>
<h3>Generating the reports</h3>
<ul>
<li>To produce the RSSAC reports 2 scripts must be run <span>(see the&nbsp;Hedgehog Tools guide for more information)</span>:
<ol>
<li><span>the&nbsp;</span><em>&lt;prefix&gt;/bin/hedgehogctl database_process_rssac_data</em><span>&nbsp;script via the Database tools component</span></li>
<li>the <em>&lt;prefix&gt;/bin/hedgehogctl rssac_generate_reports</em>&nbsp;script on the Web front-end component that will store and display the results&nbsp;</li></ol></li>
<li><span>The resulting yaml files (along with matching plot files) are stored in the directory listed in the&nbsp;</span><em>&lt;prefix&gt;/etc/hedgehog/hedgehog.yaml</em><span>&nbsp;file in the directories section for '</span><em>rssac</em><span>' (this path is auto-generated in 2.0 and should not be changed by the user).&nbsp;</span></li>
<li>The generated files are served by the Hedgehog front end and can be found by clicking on the &quot;RSSAC reports&quot; link at the bottom left of the Hedgehog webpage.&nbsp;</li></ul><ac:structured-macro ac:name="info" ac:schema-version="1" ac:macro-id="515cb692-5b1f-4ad1-8da2-56862ef2a46f"><ac:rich-text-body>
<p>Note that RSSAC reports can be run for any server, however the name of the server affects the value in the service field of the generated yaml: &nbsp;</p>
<p>If the server name is of the pattern &quot;*-root&quot; where * is a letter, then the service will be &quot;*.<a href="http://root-servers.org">root-servers.org</a>&quot;. For all other server names the service will simply be the server name.&nbsp;</p></ac:rich-text-body></ac:structured-macro>
<p>The script&nbsp;can either be run manually, or via a cron job as described in the Maintenance section of the installation guide.</p>
<h3>Notes</h3>
<p>Some notes on the RSSAC data:</p>
<ul>
<li>The DSC collector cannot collect extended RCODES and therefore Hedgehog cannot include this information in the rcode_volume report<br />&nbsp;</li>
<li>As a result of the DSC collector mechanism (i.e collecting data in 1 minute windows) the total quantity of data collected in the 'unique_sources' dataset can be significant and can result in exceptionally large database tables for this dataset. However, this 'raw' data must be uploaded to the Hedgehog database so that it can be accumulated over the 24 hour report period.<br /><br />
<ul>
<li>Once the raw data for the entire 24 hours is uploaded, the RSSAC report can be generated to create the summary figures required in the report and the 'raw' data may be discarded to save disk space (see the Maintanance section of the Installation guide for details of the script to do this).<br />&nbsp;</li>
<li>The 'Unique sources' plot in the Hedgehog GUI is generated from the summary data, not from the raw data and therefore the plot cannot be viewed until after the relevant RSSAC report has been generated via the <em>rssac_generate_reports</em> script.&nbsp;</li></ul></li></ul>
<h1>Limitations/Known issues</h1>
<p>Users should be aware of several limitations in the current version of Hedgehog as listed below. It is hoped they will be addressed in future versions</p>
<ul>
<li>&nbsp;The full list of possible plots are displayed via the drop down menu in the GUI, regardless of whether there is data in the database for the plot.<br />&nbsp;</li>
<li>
<p>There is a known issue with the SVG line plots where long legends are not wrapped properly:<br /><a href="https://code.google.com/p/google-visualization-api-issues/issues/detail?id=1536">https://code.google.com/p/google-visualization-api-issues/issues/detail?id=1536</a>&nbsp;. This affects, for example, the 'by node' plot if there are many nodes.&nbsp;<span>&nbsp;A configuration option is available (</span><em>default_interactive_plot_type</em><span>&nbsp;in&nbsp;</span><em>hedgehog_gui</em><span>.yaml) which can specify the system to produce non-SVG line plots that do not have this issue - however these plots require Flash.<br /><br /></span></p></li>
<li>No static plots are provided for the GEO plot category. Plotting of the GEO &quot;Busiest client locations by country' plot is dynamic and may be slow if there are many countries to be plotted.&nbsp;<br />&nbsp;</li>
<li>Server, node and group and instance names may only contain alphanumeric characters, full stops (.) and hyphens (-).<span>&nbsp;Country and city names can only contain&nbsp;<span>alphanumeric characters, spaces and hyphens (-)</span></span><br />&nbsp;</li>
<li>The Unique Sources plot is not available until after the RSSAC report has been generated for the date in question.&nbsp;</li></ul>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

AC_DEFUN([ACX_COMPONENTS],
[
    AC_ARG_ENABLE([web],
                AS_HELP_STRING([--disable-web],
                [Don't build the web interface - enabled by default.]),
                [AS_IF([test "x$enableval" == "xno"], [WEB=0])])
                
    AC_ARG_ENABLE([data-manager],
                AS_HELP_STRING([--disable-data-manager],
                [Don't build the data manager - enabled by default.]),
                [AS_IF([test "x$enableval" == "xno"], [DM=0])])

])
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

AC_DEFUN([ACX_POSTGRES],[
	AC_ARG_WITH([postgres], 
		[AC_HELP_STRING([--with-postgres=PATH],[specify prefix of path of PostgreSQL])],
        	[
			POSTGRES_PATH="$withval"
		],[
			POSTGRES_PATH="/usr/local"
		])

        AC_ARG_WITH([postgres-includes], 
                [AC_HELP_STRING([--with-postgres-includes=PATH],[specify include prefix of PostgreSQL])],
                [
                        POSTGRES_INCLUDE_PATH="$withval"
                ],[
                        POSTGRES_INCLUDE_PATH="/usr/include"
                ])


	AC_MSG_CHECKING(what are the postgres includes)
	POSTGRES_INCLUDES="-I$POSTGRES_INCLUDE_PATH"
	AC_MSG_RESULT($POSTGRES_INCLUDES)

	AC_MSG_CHECKING(what are the postgres libs)
	POSTGRES_LIBS="-L$POSTGRES_PATH/lib64 -L$POSTGRES_PATH/lib -lpq"
	AC_MSG_RESULT($POSTGRES_LIBS)

	tmp_CFLAGS=$CFLAGS
	tmp_LIBS=$LIBS

	CFLAGS="$CFLAGS $POSTGRES_INCLUDES"
	LIBS="$LIBS $POSTGRES_LIBS"

	AC_CHECK_LIB(pq, PQconnectdb,,[AC_MSG_ERROR([Can't find PostgreSQL])])
	LIBS=$tmp_LIBS
	CFLAGS=$tmp_CFLAGS

	AC_SUBST(POSTGRES_INCLUDES)
	AC_SUBST(POSTGRES_LIBS)
])
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

AC_DEFUN([ACX_LIBPQXX],[
	AC_ARG_WITH([libpqxx], 
		[AC_HELP_STRING([--with-libpqxx=PATH],[specify prefix of path of libpqxx])],
      	[
			LIBPQXX_PATH="$withval"
		],[
			LIBPQXX_PATH="/usr/local"
		])

	AC_MSG_CHECKING(what are the libpqxx includes)
	LIBPQXX_INCLUDES="-I$LIBPQXX_PATH/include"
	AC_MSG_RESULT($LIBPQXX_INCLUDES)

	AC_MSG_CHECKING(what are the libpqxx libs)
	LIBPQXX_LIBS="-L$LIBPQXX_PATH/lib64 -L$LIBPQXX_PATH/lib -lpqxx"
	AC_MSG_RESULT($LIBPQXX_LIBS)

        tmp_CFLAGS=$CFLAGS
        tmp_CXXFLAGS=$CXXFLAGS
	tmp_LIBS=$LIBS

	CFLAGS="$CFLAGS $LIBPQXX_INCLUDES"
	CXXFLAGS="$CXXFLAGS $LIBPQXX_INCLUDES"
	LIBS="$LIBS $LIBPQXX_LIBS"

	AC_LANG(C++)
        AC_LINK_IFELSE(
                [AC_LANG_PROGRAM([#include "pqxx/pqxx"],
                  [pqxx::connection db_conn; pqxx::work db_trans( db_conn ); pqxx::pipeline db_pipe(db_trans)])],
                  ,
                  [AC_MSG_ERROR([Can't find libpqxx])])

	LIBS=$tmp_LIBS
	CFLAGS=$tmp_CFLAGS
        CXXFLAGS=$tmp_CXXFLAGS

	AC_SUBST(LIBPQXX_INCLUDES)
	AC_SUBST(LIBPQXX_LIBS)
])
# boost.m4: Locate Boost headers and libraries for autoconf-based projects.
# Copyright (C) 2007-2011, 2014  Benoit Sigoure <tsuna@lrde.epita.fr>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Additional permission under section 7 of the GNU General Public
# License, version 3 ("GPLv3"):
#
# If you convey this file as part of a work that contains a
# configuration script generated by Autoconf, you may do so under
# terms of your choice.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

m4_define([_BOOST_SERIAL], [m4_translit([
# serial 24
], [#
], [])])

# Original sources can be found at http://github.com/tsuna/boost.m4
# You can fetch the latest version of the script by doing:
#   wget http://github.com/tsuna/boost.m4/raw/master/build-aux/boost.m4

# ------ #
# README #
# ------ #

# This file provides several macros to use the various Boost libraries.
# The first macro is BOOST_REQUIRE.  It will simply check if it's possible to
# find the Boost headers of a given (optional) minimum version and it will
# define BOOST_CPPFLAGS accordingly.  It will add an option --with-boost to
# your configure so that users can specify non standard locations.
# If the user's environment contains BOOST_ROOT and --with-boost was not
# specified, --with-boost=$BOOST_ROOT is implicitly used.
# For more README and documentation, go to http://github.com/tsuna/boost.m4
# Note: THESE MACROS ASSUME THAT YOU USE LIBTOOL.  If you don't, don't worry,
# simply read the README, it will show you what to do step by step.

m4_pattern_forbid([^_?(BOOST|Boost)_])


# _BOOST_SED_CPP(SED-PROGRAM, PROGRAM,
#                [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND])
# --------------------------------------------------------
# Same as AC_EGREP_CPP, but leave the result in conftest.i.
#
# SED-PROGRAM is *not* overquoted, as in AC_EGREP_CPP.  It is expanded
# in double-quotes, so escape your double quotes.
#
# It could be useful to turn this into a macro which extracts the
# value of any macro.
m4_define([_BOOST_SED_CPP],
[AC_LANG_PUSH([C++])dnl
AC_LANG_PREPROC_REQUIRE()dnl
AC_REQUIRE([AC_PROG_SED])dnl
AC_LANG_CONFTEST([AC_LANG_SOURCE([[$2]])])
AS_IF([dnl eval is necessary to expand ac_cpp.
dnl Ultrix and Pyramid sh refuse to redirect output of eval, so use subshell.
dnl Beware of Windows end-of-lines, for instance if we are running
dnl some Windows programs under Wine.  In that case, boost/version.hpp
dnl is certainly using "\r\n", but the regular Unix shell will only
dnl strip `\n' with backquotes, not the `\r'.  This results in
dnl boost_cv_lib_version='1_37\r' for instance, which breaks
dnl everything else.
dnl Cannot use 'dnl' after [$4] because a trailing dnl may break AC_CACHE_CHECK
dnl
dnl Beware that GCC 5, when expanding macros, may embed # line directives
dnl a within single line:
dnl
dnl # 1 "conftest.cc"
dnl # 1 "<built-in>"
dnl # 1 "<command-line>"
dnl # 1 "conftest.cc"
dnl # 1 "/opt/local/include/boost/version.hpp" 1 3
dnl # 2 "conftest.cc" 2
dnl boost-lib-version =
dnl # 2 "conftest.cc" 3
dnl                    "1_56"
dnl
dnl So get rid of the # lines, and glue the remaining ones together.
(eval "$ac_cpp conftest.$ac_ext") 2>&AS_MESSAGE_LOG_FD |
  grep -v '#' |
  tr -d '\r' |
  tr -s '\n' ' ' |
  $SED -n -e "$1" >conftest.i 2>&1],
  [$3],
  [$4])
rm -rf conftest*
AC_LANG_POP([C++])dnl
])# _BOOST_SED_CPP



# BOOST_REQUIRE([VERSION], [ACTION-IF-NOT-FOUND])
# -----------------------------------------------
# Look for Boost.  If version is given, it must either be a literal of the form
# "X.Y.Z" where X, Y and Z are integers (the ".Z" part being optional) or a
# variable "$var".
# Defines the value BOOST_CPPFLAGS.  This macro only checks for headers with
# the required version, it does not check for any of the Boost libraries.
# On # success, defines HAVE_BOOST.  On failure, calls the optional
# ACTION-IF-NOT-FOUND action if one was supplied.
# Otherwise aborts with an error message.
AC_DEFUN([BOOST_REQUIRE],
[AC_REQUIRE([AC_PROG_CXX])dnl
AC_REQUIRE([AC_PROG_GREP])dnl
echo "$as_me: this is boost.m4[]_BOOST_SERIAL" >&AS_MESSAGE_LOG_FD
boost_save_IFS=$IFS
boost_version_req=$1
IFS=.
set x $boost_version_req 0 0 0
IFS=$boost_save_IFS
shift
boost_version_req=`expr "$[1]" '*' 100000 + "$[2]" '*' 100 + "$[3]"`
boost_version_req_string=$[1].$[2].$[3]
AC_ARG_WITH([boost],
   [AS_HELP_STRING([--with-boost=DIR],
                   [prefix of Boost $1 @<:@guess@:>@])])dnl
AC_ARG_VAR([BOOST_ROOT],[Location of Boost installation])dnl
# If BOOST_ROOT is set and the user has not provided a value to
# --with-boost, then treat BOOST_ROOT as if it the user supplied it.
if test x"$BOOST_ROOT" != x; then
  if test x"$with_boost" = x; then
    AC_MSG_NOTICE([Detected BOOST_ROOT; continuing with --with-boost=$BOOST_ROOT])
    with_boost=$BOOST_ROOT
  else
    AC_MSG_NOTICE([Detected BOOST_ROOT=$BOOST_ROOT, but overridden by --with-boost=$with_boost])
  fi
fi
AC_SUBST([DISTCHECK_CONFIGURE_FLAGS],
         ["$DISTCHECK_CONFIGURE_FLAGS '--with-boost=$with_boost'"])dnl
boost_save_CPPFLAGS=$CPPFLAGS
  AC_CACHE_CHECK([for Boost headers version >= $boost_version_req_string],
    [boost_cv_inc_path],
    [boost_cv_inc_path=no
AC_LANG_PUSH([C++])dnl
m4_pattern_allow([^BOOST_VERSION$])dnl
    AC_LANG_CONFTEST([AC_LANG_PROGRAM([[#include <boost/version.hpp>
#if !defined BOOST_VERSION
# error BOOST_VERSION is not defined
#elif BOOST_VERSION < $boost_version_req
# error Boost headers version < $boost_version_req
#endif
]])])
    # If the user provided a value to --with-boost, use it and only it.
    case $with_boost in #(
      ''|yes) set x '' /opt/local/include /usr/local/include /opt/include \
                 /usr/include C:/Boost/include;; #(
      *)      set x "$with_boost/include" "$with_boost";;
    esac
    shift
    for boost_dir
    do
    # Without --layout=system, Boost (or at least some versions) installs
    # itself in <prefix>/include/boost-<version>.  This inner loop helps to
    # find headers in such directories.
    #
    # Any ${boost_dir}/boost-x_xx directories are searched in reverse version
    # order followed by ${boost_dir}.  The final '.' is a sentinel for
    # searching $boost_dir" itself.  Entries are whitespace separated.
    #
    # I didn't indent this loop on purpose (to avoid over-indented code)
    boost_layout_system_search_list=`cd "$boost_dir" 2>/dev/null \
        && ls -1 | "${GREP}" '^boost-' | sort -rn -t- -k2 \
        && echo .`
    for boost_inc in $boost_layout_system_search_list
    do
      if test x"$boost_inc" != x.; then
        boost_inc="$boost_dir/$boost_inc"
      else
        boost_inc="$boost_dir" # Uses sentinel in boost_layout_system_search_list
      fi
      if test x"$boost_inc" != x; then
        # We are going to check whether the version of Boost installed
        # in $boost_inc is usable by running a compilation that
        # #includes it.  But if we pass a -I/some/path in which Boost
        # is not installed, the compiler will just skip this -I and
        # use other locations (either from CPPFLAGS, or from its list
        # of system include directories).  As a result we would use
        # header installed on the machine instead of the /some/path
        # specified by the user.  So in that precise case (trying
        # $boost_inc), make sure the version.hpp exists.
        #
        # Use test -e as there can be symlinks.
        test -e "$boost_inc/boost/version.hpp" || continue
        CPPFLAGS="$CPPFLAGS -I$boost_inc"
      fi
      AC_COMPILE_IFELSE([], [boost_cv_inc_path=yes], [boost_cv_version=no])
      if test x"$boost_cv_inc_path" = xyes; then
        if test x"$boost_inc" != x; then
          boost_cv_inc_path=$boost_inc
        fi
        break 2
      fi
    done
    done
AC_LANG_POP([C++])dnl
    ])
    case $boost_cv_inc_path in #(
      no)
        boost_errmsg="cannot find Boost headers version >= $boost_version_req_string"
        m4_if([$2], [],  [AC_MSG_ERROR([$boost_errmsg])],
                        [AC_MSG_NOTICE([$boost_errmsg])])
        $2
        ;;#(
      yes)
        BOOST_CPPFLAGS=
        ;;#(
      *)
        AC_SUBST([BOOST_CPPFLAGS], ["-I$boost_cv_inc_path"])dnl
        ;;
    esac
  if test x"$boost_cv_inc_path" != xno; then
  AC_DEFINE([HAVE_BOOST], [1],
            [Defined if the requested minimum BOOST version is satisfied])
  AC_CACHE_CHECK([for Boost's header version],
    [boost_cv_lib_version],
    [m4_pattern_allow([^BOOST_LIB_VERSION$])dnl
     _BOOST_SED_CPP([[/^boost-lib-version = /{s///;s/[\" ]//g;p;q;}]],
                    [#include <boost/version.hpp>
boost-lib-version = BOOST_LIB_VERSION],
    [boost_cv_lib_version=`cat conftest.i`])])
    # e.g. "134" for 1_34_1 or "135" for 1_35
    boost_major_version=`echo "$boost_cv_lib_version" | sed 's/_//;s/_.*//'`
    case $boost_major_version in #(
      '' | *[[!0-9]]*)
        AC_MSG_ERROR([invalid value: boost_major_version='$boost_major_version'])
        ;;
    esac
fi
CPPFLAGS=$boost_save_CPPFLAGS
])# BOOST_REQUIRE


# BOOST_STATIC()
# --------------
# Add the "--enable-static-boost" configure argument. If this argument is given
# on the command line, static versions of the libraries will be looked up.
AC_DEFUN([BOOST_STATIC],
  [AC_ARG_ENABLE([static-boost],
     [AS_HELP_STRING([--enable-static-boost],
               [Prefer the static boost libraries over the shared ones [no]])],
     [enable_static_boost=yes],
     [enable_static_boost=no])])# BOOST_STATIC


# BOOST_FIND_HEADER([HEADER-NAME], [ACTION-IF-NOT-FOUND], [ACTION-IF-FOUND])
# --------------------------------------------------------------------------
# Wrapper around AC_CHECK_HEADER for Boost headers.  Useful to check for
# some parts of the Boost library which are only made of headers and don't
# require linking (such as Boost.Foreach).
#
# Default ACTION-IF-NOT-FOUND: Fail with a fatal error unless Boost couldn't be
# found in the first place, in which case by default a notice is issued to the
# user.  Presumably if we haven't died already it's because it's OK to not have
# Boost, which is why only a notice is issued instead of a hard error.
#
# Default ACTION-IF-FOUND: define the preprocessor symbol HAVE_<HEADER-NAME> in
# case of success # (where HEADER-NAME is written LIKE_THIS, e.g.,
# HAVE_BOOST_FOREACH_HPP).
AC_DEFUN([BOOST_FIND_HEADER],
[AC_REQUIRE([BOOST_REQUIRE])dnl
if test x"$boost_cv_inc_path" = xno; then
  m4_default([$2], [AC_MSG_NOTICE([Boost not available, not searching for $1])])
else
AC_LANG_PUSH([C++])dnl
boost_save_CPPFLAGS=$CPPFLAGS
CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS"
AC_CHECK_HEADER([$1],
  [m4_default([$3], [AC_DEFINE(AS_TR_CPP([HAVE_$1]), [1],
                               [Define to 1 if you have <$1>])])],
  [m4_default([$2], [AC_MSG_ERROR([cannot find $1])])])
CPPFLAGS=$boost_save_CPPFLAGS
AC_LANG_POP([C++])dnl
fi
])# BOOST_FIND_HEADER


# BOOST_FIND_LIBS([COMPONENT-NAME], [CANDIDATE-LIB-NAMES],
#                 [PREFERRED-RT-OPT], [HEADER-NAME], [CXX-TEST],
#                 [CXX-PROLOGUE])
# --------------------------------------------------------------
# Look for the Boost library COMPONENT-NAME (e.g., `thread', for
# libboost_thread) under the possible CANDIDATE-LIB-NAMES (e.g.,
# "thread_win32 thread").  Check that HEADER-NAME works and check that
# libboost_LIB-NAME can link with the code CXX-TEST.  The optional
# argument CXX-PROLOGUE can be used to include some C++ code before
# the `main' function.
#
# Invokes BOOST_FIND_HEADER([HEADER-NAME]) (see above).
#
# Boost libraries typically come compiled with several flavors (with different
# runtime options) so PREFERRED-RT-OPT is the preferred suffix.  A suffix is one
# or more of the following letters: sgdpn (in that order).  s = static
# runtime, d = debug build, g = debug/diagnostic runtime, p = STLPort build,
# n = (unsure) STLPort build without iostreams from STLPort (it looks like `n'
# must always be used along with `p').  Additionally, PREFERRED-RT-OPT can
# start with `mt-' to indicate that there is a preference for multi-thread
# builds.  Some sample values for PREFERRED-RT-OPT: (nothing), mt, d, mt-d, gdp
# ...  If you want to make sure you have a specific version of Boost
# (eg, >= 1.33) you *must* invoke BOOST_REQUIRE before this macro.
AC_DEFUN([BOOST_FIND_LIBS],
[AC_REQUIRE([BOOST_REQUIRE])dnl
AC_REQUIRE([_BOOST_FIND_COMPILER_TAG])dnl
AC_REQUIRE([BOOST_STATIC])dnl
AC_REQUIRE([_BOOST_GUESS_WHETHER_TO_USE_MT])dnl
if test x"$boost_cv_inc_path" = xno; then
  AC_MSG_NOTICE([Boost not available, not searching for the Boost $1 library])
else
dnl The else branch is huge and wasn't intended on purpose.
AC_LANG_PUSH([C++])dnl
AS_VAR_PUSHDEF([Boost_lib], [boost_cv_lib_$1])dnl
AS_VAR_PUSHDEF([Boost_lib_LDFLAGS], [boost_cv_lib_$1_LDFLAGS])dnl
AS_VAR_PUSHDEF([Boost_lib_LDPATH], [boost_cv_lib_$1_LDPATH])dnl
AS_VAR_PUSHDEF([Boost_lib_LIBS], [boost_cv_lib_$1_LIBS])dnl
BOOST_FIND_HEADER([$4])
boost_save_CPPFLAGS=$CPPFLAGS
CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS"
AC_CACHE_CHECK([for the Boost $1 library], [Boost_lib],
               [_BOOST_FIND_LIBS($@)])
case $Boost_lib in #(
  (no) _AC_MSG_LOG_CONFTEST
    AC_MSG_ERROR([cannot find the flags to link with Boost $1])
    ;;
esac
AC_SUBST(AS_TR_CPP([BOOST_$1_LDFLAGS]), [$Boost_lib_LDFLAGS])dnl
AC_SUBST(AS_TR_CPP([BOOST_$1_LDPATH]), [$Boost_lib_LDPATH])dnl
AC_SUBST([BOOST_LDPATH], [$Boost_lib_LDPATH])dnl
AC_SUBST(AS_TR_CPP([BOOST_$1_LIBS]), [$Boost_lib_LIBS])dnl
CPPFLAGS=$boost_save_CPPFLAGS
AS_VAR_POPDEF([Boost_lib])dnl
AS_VAR_POPDEF([Boost_lib_LDFLAGS])dnl
AS_VAR_POPDEF([Boost_lib_LDPATH])dnl
AS_VAR_POPDEF([Boost_lib_LIBS])dnl
AC_LANG_POP([C++])dnl
fi
])


# BOOST_FIND_LIB([LIB-NAME],
#                [PREFERRED-RT-OPT], [HEADER-NAME], [CXX-TEST],
#                [CXX-PROLOGUE])
# --------------------------------------------------------------
# Backward compatibility wrapper for BOOST_FIND_LIBS.
AC_DEFUN([BOOST_FIND_LIB],
[BOOST_FIND_LIBS([$1], $@)])


# _BOOST_FIND_LIBS([LIB-NAME], [CANDIDATE-LIB-NAMES],
#                 [PREFERRED-RT-OPT], [HEADER-NAME], [CXX-TEST],
#                 [CXX-PROLOGUE])
# --------------------------------------------------------------
# Real implementation of BOOST_FIND_LIBS: rely on these local macros:
# Boost_lib, Boost_lib_LDFLAGS, Boost_lib_LDPATH, Boost_lib_LIBS
#
# The algorithm is as follows: first look for a given library name
# according to the user's PREFERRED-RT-OPT.  For each library name, we
# prefer to use the ones that carry the tag (toolset name).  Each
# library is searched through the various standard paths were Boost is
# usually installed.  If we can't find the standard variants, we try
# to enforce -mt (for instance on MacOSX, libboost_thread.dylib
# doesn't exist but there's -obviously- libboost_thread-mt.dylib).
AC_DEFUN([_BOOST_FIND_LIBS],
[Boost_lib=no
  case "$3" in #(
    (mt | mt-) boost_mt=-mt; boost_rtopt=;; #(
    (mt* | mt-*) boost_mt=-mt; boost_rtopt=`expr "X$3" : 'Xmt-*\(.*\)'`;; #(
    (*) boost_mt=; boost_rtopt=$3;;
  esac
  if test $enable_static_boost = yes; then
    boost_rtopt="s$boost_rtopt"
  fi
  # Find the proper debug variant depending on what we've been asked to find.
  case $boost_rtopt in #(
    (*d*) boost_rt_d=$boost_rtopt;; #(
    (*[[sgpn]]*) # Insert the `d' at the right place (in between `sg' and `pn')
      boost_rt_d=`echo "$boost_rtopt" | sed 's/\(s*g*\)\(p*n*\)/\1\2/'`;; #(
    (*) boost_rt_d='-d';;
  esac
  # If the PREFERRED-RT-OPT are not empty, prepend a `-'.
  test -n "$boost_rtopt" && boost_rtopt="-$boost_rtopt"
  $boost_guess_use_mt && boost_mt=-mt
  # Look for the abs path the static archive.
  # $libext is computed by Libtool but let's make sure it's non empty.
  test -z "$libext" &&
    AC_MSG_ERROR([the libext variable is empty, did you invoke Libtool?])
  boost_save_ac_objext=$ac_objext
  # Generate the test file.
  AC_LANG_CONFTEST([AC_LANG_PROGRAM([#include <$4>
$6], [$5])])
dnl Optimization hacks: compiling C++ is slow, especially with Boost.  What
dnl we're trying to do here is guess the right combination of link flags
dnl (LIBS / LDFLAGS) to use a given library.  This can take several
dnl iterations before it succeeds and is thus *very* slow.  So what we do
dnl instead is that we compile the code first (and thus get an object file,
dnl typically conftest.o).  Then we try various combinations of link flags
dnl until we succeed to link conftest.o in an executable.  The problem is
dnl that the various TRY_LINK / COMPILE_IFELSE macros of Autoconf always
dnl remove all the temporary files including conftest.o.  So the trick here
dnl is to temporarily change the value of ac_objext so that conftest.o is
dnl preserved accross tests.  This is obviously fragile and I will burn in
dnl hell for not respecting Autoconf's documented interfaces, but in the
dnl mean time, it optimizes the macro by a factor of 5 to 30.
dnl Another small optimization: the first argument of AC_COMPILE_IFELSE left
dnl empty because the test file is generated only once above (before we
dnl start the for loops).
  AC_COMPILE_IFELSE([],
    [ac_objext=do_not_rm_me_plz],
    [AC_MSG_ERROR([cannot compile a test that uses Boost $1])])
  ac_objext=$boost_save_ac_objext
  boost_failed_libs=
# Don't bother to ident the following nested for loops, only the 2
# innermost ones matter.
for boost_lib_ in $2; do
for boost_tag_ in -$boost_cv_lib_tag ''; do
for boost_ver_ in -$boost_cv_lib_version ''; do
for boost_mt_ in $boost_mt -mt ''; do
for boost_rtopt_ in $boost_rtopt '' -d; do
  for boost_lib in \
    boost_$boost_lib_$boost_tag_$boost_mt_$boost_rtopt_$boost_ver_ \
    boost_$boost_lib_$boost_tag_$boost_rtopt_$boost_ver_ \
    boost_$boost_lib_$boost_tag_$boost_mt_$boost_ver_ \
    boost_$boost_lib_$boost_tag_$boost_ver_
  do
    # Avoid testing twice the same lib
    case $boost_failed_libs in #(
      (*@$boost_lib@*) continue;;
    esac
    # If with_boost is empty, we'll search in /lib first, which is not quite
    # right so instead we'll try to a location based on where the headers are.
    boost_tmp_lib=$with_boost
    test x"$with_boost" = x && boost_tmp_lib=${boost_cv_inc_path%/include}
    for boost_ldpath in "$boost_tmp_lib/lib" '' \
             /opt/local/lib* /usr/local/lib* /opt/lib* /usr/lib* \
             "$with_boost" C:/Boost/lib /lib*
    do
      # Don't waste time with directories that don't exist.
      if test x"$boost_ldpath" != x && test ! -e "$boost_ldpath"; then
        continue
      fi
      boost_save_LDFLAGS=$LDFLAGS
      # Are we looking for a static library?
      case $boost_ldpath:$boost_rtopt_ in #(
        (*?*:*s*) # Yes (Non empty boost_ldpath + s in rt opt)
          Boost_lib_LIBS="$boost_ldpath/lib$boost_lib.$libext"
          test -e "$Boost_lib_LIBS" || continue;; #(
        (*) # No: use -lboost_foo to find the shared library.
          Boost_lib_LIBS="-l$boost_lib";;
      esac
      boost_save_LIBS=$LIBS
      LIBS="$Boost_lib_LIBS $LIBS"
      test x"$boost_ldpath" != x && LDFLAGS="$LDFLAGS -L$boost_ldpath"
dnl First argument of AC_LINK_IFELSE left empty because the test file is
dnl generated only once above (before we start the for loops).
      _BOOST_AC_LINK_IFELSE([],
                            [Boost_lib=yes], [Boost_lib=no])
      ac_objext=$boost_save_ac_objext
      LDFLAGS=$boost_save_LDFLAGS
      LIBS=$boost_save_LIBS
      if test x"$Boost_lib" = xyes; then
        test x"$boost_ldpath" != x &&
          Boost_lib_LDFLAGS="-L$boost_ldpath $boost_cv_rpath_link_ldflag$boost_ldpath"
        Boost_lib_LDPATH="$boost_ldpath"
        break 7
      else
        boost_failed_libs="$boost_failed_libs@$boost_lib@"
      fi
    done
  done
done
done
done
done
done # boost_lib_
rm -f conftest.$ac_objext
])



# --------------------------------------- #
# Checks for the various Boost libraries. #
# --------------------------------------- #

# List of boost libraries: http://www.boost.org/libs/libraries.htm
# The page http://beta.boost.org/doc/libs is useful: it gives the first release
# version of each library (among other things).

# BOOST_DEFUN(LIBRARY, CODE)
# --------------------------
# Define BOOST_<LIBRARY-UPPERCASE> as a macro that runs CODE.
#
# Use indir to avoid the warning on underquoted macro name given to AC_DEFUN.
m4_define([BOOST_DEFUN],
[m4_indir([AC_DEFUN],
          m4_toupper([BOOST_$1]),
[m4_pushdef([BOOST_Library], [$1])dnl
$2
m4_popdef([BOOST_Library])dnl
])
])

# BOOST_ARRAY()
# -------------
# Look for Boost.Array
BOOST_DEFUN([Array],
[BOOST_FIND_HEADER([boost/array.hpp])])


# BOOST_ASIO()
# ------------
# Look for Boost.Asio (new in Boost 1.35).
BOOST_DEFUN([Asio],
[AC_REQUIRE([BOOST_SYSTEM])dnl
BOOST_FIND_HEADER([boost/asio.hpp])])


# BOOST_BIND()
# ------------
# Look for Boost.Bind.
BOOST_DEFUN([Bind],
[BOOST_FIND_HEADER([boost/bind.hpp])])


# BOOST_CHRONO()
# --------------
# Look for Boost.Chrono.
BOOST_DEFUN([Chrono],
[# Do we have to check for Boost.System?  This link-time dependency was
# added as of 1.35.0.  If we have a version <1.35, we must not attempt to
# find Boost.System as it didn't exist by then.
if test $boost_major_version -ge 135; then
  BOOST_SYSTEM([$1])
fi # end of the Boost.System check.
boost_filesystem_save_LIBS=$LIBS
boost_filesystem_save_LDFLAGS=$LDFLAGS
m4_pattern_allow([^BOOST_SYSTEM_(LIBS|LDFLAGS)$])dnl
LIBS="$LIBS $BOOST_SYSTEM_LIBS"
LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS"
BOOST_FIND_LIB([chrono], [$1],
                [boost/chrono.hpp],
                [boost::chrono::thread_clock d;])
if test $enable_static_boost = yes && test $boost_major_version -ge 135; then
  BOOST_CHRONO_LIBS="$BOOST_CHRONO_LIBS $BOOST_SYSTEM_LIBS"
fi
LIBS=$boost_filesystem_save_LIBS
LDFLAGS=$boost_filesystem_save_LDFLAGS
])# BOOST_CHRONO


# BOOST_CONTEXT([PREFERRED-RT-OPT])
# -----------------------------------
# Look for Boost.Context.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.  This library was introduced in Boost
# 1.51.0
BOOST_DEFUN([Context],
[BOOST_FIND_LIB([context], [$1],
                [boost/context/all.hpp],[[
// creates a stack
void * stack_pointer = new void*[4096];
std::size_t const size = sizeof(void*[4096]);

// context fc uses f() as context function
// fcontext_t is placed on top of context stack
// a pointer to fcontext_t is returned
fc = ctx::make_fcontext(stack_pointer, size, f);
return ctx::jump_fcontext(&fcm, fc, 3) == 6;]],[dnl
namespace ctx = boost::context;
// context
static ctx::fcontext_t fcm, *fc;
// context-function
static void f(intptr_t i) {
    ctx::jump_fcontext(fc, &fcm, i * 2);
}])
])# BOOST_CONTEXT


# BOOST_CONVERSION()
# ------------------
# Look for Boost.Conversion (cast / lexical_cast)
BOOST_DEFUN([Conversion],
[BOOST_FIND_HEADER([boost/cast.hpp])
BOOST_FIND_HEADER([boost/lexical_cast.hpp])
])# BOOST_CONVERSION


# BOOST_COROUTINE([PREFERRED-RT-OPT])
# -----------------------------------
# Look for Boost.Coroutine.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.  This library was introduced in Boost
# 1.53.0
BOOST_DEFUN([Coroutine],
[
boost_coroutine_save_LIBS=$LIBS
boost_coroutine_save_LDFLAGS=$LDFLAGS
# Link-time dependency from coroutine to context
BOOST_CONTEXT([$1])
# Starting from Boost 1.55 a dependency on Boost.System is added
if test $boost_major_version -ge 155; then
  BOOST_SYSTEM([$1])
fi
m4_pattern_allow([^BOOST_(CONTEXT|SYSTEM)_(LIBS|LDFLAGS)])
LIBS="$LIBS $BOOST_CONTEXT_LIBS $BOOST_SYSTEM_LIBS"
LDFLAGS="$LDFLAGS $BOOST_CONTEXT_LDFLAGS"

BOOST_FIND_LIB([coroutine], [$1],
                [boost/coroutine/coroutine.hpp],
                [boost::coroutines::coroutine< int(int) > coro; coro.empty();])

# Link-time dependency from coroutine to context, existed only in 1.53, in 1.54
# coroutine doesn't use context from its headers but from its library.
if test $boost_major_version -eq 153 || test $enable_static_boost = yes && test $boost_major_version -ge 154; then
  BOOST_COROUTINE_LIBS="$BOOST_COROUTINE_LIBS $BOOST_CONTEXT_LIBS"
  BOOST_COROUTINE_LDFLAGS="$BOOST_COROUTINE_LDFLAGS $BOOST_CONTEXT_LDFLAGS"
fi
if test $enable_static_boost = yes && test $boost_major_version -ge 155; then
  BOOST_COROUTINE_LIBS="$BOOST_COROUTINE_LIBS $BOOST_SYSTEM_LIBS"
  BOOST_COROUTINE_LDFLAGS="$BOOST_COROUTINE_LDFLAGS $BOOST_SYSTEM_LDFLAGS"
fi
LIBS=$boost_coroutine_save_LIBS
LDFLAGS=$boost_coroutine_save_LDFLAGS
])# BOOST_COROUTINE


# BOOST_CRC()
# -----------
# Look for Boost.CRC
BOOST_DEFUN([CRC],
[BOOST_FIND_HEADER([boost/crc.hpp])
])# BOOST_CRC


# BOOST_DATE_TIME([PREFERRED-RT-OPT])
# -----------------------------------
# Look for Boost.Date_Time.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Date_Time],
[BOOST_FIND_LIB([date_time], [$1],
                [boost/date_time/posix_time/posix_time.hpp],
                [boost::posix_time::ptime t;])
])# BOOST_DATE_TIME


# BOOST_FILESYSTEM([PREFERRED-RT-OPT])
# ------------------------------------
# Look for Boost.Filesystem.  For the documentation of PREFERRED-RT-OPT, see
# the documentation of BOOST_FIND_LIB above.
# Do not check for boost/filesystem.hpp because this file was introduced in
# 1.34.
BOOST_DEFUN([Filesystem],
[# Do we have to check for Boost.System?  This link-time dependency was
# added as of 1.35.0.  If we have a version <1.35, we must not attempt to
# find Boost.System as it didn't exist by then.
if test $boost_major_version -ge 135; then
  BOOST_SYSTEM([$1])
fi # end of the Boost.System check.
boost_filesystem_save_LIBS=$LIBS
boost_filesystem_save_LDFLAGS=$LDFLAGS
m4_pattern_allow([^BOOST_SYSTEM_(LIBS|LDFLAGS)$])dnl
LIBS="$LIBS $BOOST_SYSTEM_LIBS"
LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS"
BOOST_FIND_LIB([filesystem], [$1],
                [boost/filesystem/path.hpp], [boost::filesystem::path p;])
if test $enable_static_boost = yes && test $boost_major_version -ge 135; then
  BOOST_FILESYSTEM_LIBS="$BOOST_FILESYSTEM_LIBS $BOOST_SYSTEM_LIBS"
fi
LIBS=$boost_filesystem_save_LIBS
LDFLAGS=$boost_filesystem_save_LDFLAGS
])# BOOST_FILESYSTEM


# BOOST_FLYWEIGHT()
# -----------------
# Look for Boost.Flyweight.
BOOST_DEFUN([Flyweight],
[dnl There's a hidden dependency on pthreads.
AC_REQUIRE([_BOOST_PTHREAD_FLAG])dnl
BOOST_FIND_HEADER([boost/flyweight.hpp])
AC_SUBST([BOOST_FLYWEIGHT_LIBS], [$boost_cv_pthread_flag])
])


# BOOST_FOREACH()
# ---------------
# Look for Boost.Foreach.
BOOST_DEFUN([Foreach],
[BOOST_FIND_HEADER([boost/foreach.hpp])])


# BOOST_FORMAT()
# --------------
# Look for Boost.Format.
# Note: we can't check for boost/format/format_fwd.hpp because the header isn't
# standalone.  It can't be compiled because it triggers the following error:
# boost/format/detail/config_macros.hpp:88: error: 'locale' in namespace 'std'
#                                                  does not name a type
BOOST_DEFUN([Format],
[BOOST_FIND_HEADER([boost/format.hpp])])


# BOOST_FUNCTION()
# ----------------
# Look for Boost.Function
BOOST_DEFUN([Function],
[BOOST_FIND_HEADER([boost/function.hpp])])


# BOOST_GEOMETRY()
# ----------------
# Look for Boost.Geometry (new since 1.47.0).
BOOST_DEFUN([Geometry],
[BOOST_FIND_HEADER([boost/geometry.hpp])
])# BOOST_GEOMETRY


# BOOST_GRAPH([PREFERRED-RT-OPT])
# -------------------------------
# Look for Boost.Graphs.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Graph],
[BOOST_FIND_LIB([graph], [$1],
                [boost/graph/adjacency_list.hpp], [boost::adjacency_list<> g;])
])# BOOST_GRAPH


# BOOST_IOSTREAMS([PREFERRED-RT-OPT])
# -----------------------------------
# Look for Boost.IOStreams.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([IOStreams],
[BOOST_FIND_LIB([iostreams], [$1],
                [boost/iostreams/device/file_descriptor.hpp],
                [boost::iostreams::file_descriptor fd; fd.close();])
])# BOOST_IOSTREAMS


# BOOST_HASH()
# ------------
# Look for Boost.Functional/Hash
BOOST_DEFUN([Hash],
[BOOST_FIND_HEADER([boost/functional/hash.hpp])])


# BOOST_LAMBDA()
# --------------
# Look for Boost.Lambda
BOOST_DEFUN([Lambda],
[BOOST_FIND_HEADER([boost/lambda/lambda.hpp])])


# BOOST_LOCALE()
# --------------
# Look for Boost.Locale
BOOST_DEFUN([Locale],
[BOOST_FIND_LIB([locale], [$1],
    [boost/locale.hpp],
    [[boost::locale::generator gen; std::locale::global(gen(""));]])
])# BOOST_LOCALE

# BOOST_LOG([PREFERRED-RT-OPT])
# -----------------------------
# Look for Boost.Log.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Log],
[BOOST_FIND_LIB([log], [$1],
    [boost/log/core/core.hpp],
    [boost::log::attribute a; a.get_value();])
])# BOOST_LOG


# BOOST_LOG_SETUP([PREFERRED-RT-OPT])
# -----------------------------------
# Look for Boost.Log.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Log_Setup],
[AC_REQUIRE([BOOST_LOG])dnl
BOOST_FIND_LIB([log_setup], [$1],
    [boost/log/utility/setup/from_settings.hpp],
    [boost::log::basic_settings<char> bs; bs.empty();])
])# BOOST_LOG_SETUP


# BOOST_MATH()
# ------------
# Look for Boost.Math
# TODO: This library isn't header-only but it comes in multiple different
# flavors that don't play well with BOOST_FIND_LIB (e.g, libboost_math_c99,
# libboost_math_c99f, libboost_math_c99l, libboost_math_tr1,
# libboost_math_tr1f, libboost_math_tr1l).  This macro must be fixed to do the
# right thing anyway.
BOOST_DEFUN([Math],
[BOOST_FIND_HEADER([boost/math/special_functions.hpp])])


# BOOST_MPI([PREFERRED-RT-OPT])
# -------------------------------
# Look for Boost MPI.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.  Uses MPICXX variable if it is
# set, otherwise tries CXX
#
BOOST_DEFUN([MPI],
[boost_save_CXX=${CXX}
boost_save_CXXCPP=${CXXCPP}
if test x"${MPICXX}" != x; then
  CXX=${MPICXX}
  CXXCPP="${MPICXX} -E"
fi
BOOST_FIND_LIB([mpi], [$1],
               [boost/mpi.hpp],
               [int argc = 0;
                char **argv = 0;
                boost::mpi::environment env(argc,argv);])
CXX=${boost_save_CXX}
CXXCPP=${boost_save_CXXCPP}
])# BOOST_MPI


# BOOST_MULTIARRAY()
# ------------------
# Look for Boost.MultiArray
BOOST_DEFUN([MultiArray],
[BOOST_FIND_HEADER([boost/multi_array.hpp])])


# BOOST_NUMERIC_UBLAS()
# --------------------------
# Look for Boost.NumericUblas (Basic Linear Algebra)
BOOST_DEFUN([Numeric_Ublas],
[BOOST_FIND_HEADER([boost/numeric/ublas/vector.hpp])
])# BOOST_NUMERIC_UBLAS


# BOOST_NUMERIC_CONVERSION()
# --------------------------
# Look for Boost.NumericConversion (policy-based numeric conversion)
BOOST_DEFUN([Numeric_Conversion],
[BOOST_FIND_HEADER([boost/numeric/conversion/converter.hpp])
])# BOOST_NUMERIC_CONVERSION


# BOOST_OPTIONAL()
# ----------------
# Look for Boost.Optional
BOOST_DEFUN([Optional],
[BOOST_FIND_HEADER([boost/optional.hpp])])


# BOOST_PREPROCESSOR()
# --------------------
# Look for Boost.Preprocessor
BOOST_DEFUN([Preprocessor],
[BOOST_FIND_HEADER([boost/preprocessor/repeat.hpp])])


# BOOST_RANGE()
# --------------------
# Look for Boost.Range
BOOST_DEFUN([Range],
[BOOST_FIND_HEADER([boost/range/adaptors.hpp])])

# BOOST_UNORDERED()
# -----------------
# Look for Boost.Unordered
BOOST_DEFUN([Unordered],
[BOOST_FIND_HEADER([boost/unordered_map.hpp])])


# BOOST_UUID()
# ------------
# Look for Boost.Uuid
BOOST_DEFUN([Uuid],
[BOOST_FIND_HEADER([boost/uuid/uuid.hpp])])


# BOOST_PROGRAM_OPTIONS([PREFERRED-RT-OPT])
# -----------------------------------------
# Look for Boost.Program_options.  For the documentation of PREFERRED-RT-OPT,
# see the documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Program_Options],
[BOOST_FIND_LIB([program_options], [$1],
                [boost/program_options.hpp],
                [boost::program_options::options_description d("test");])
])# BOOST_PROGRAM_OPTIONS



# _BOOST_PYTHON_CONFIG(VARIABLE, FLAG)
# ------------------------------------
# Save VARIABLE, and define it via `python-config --FLAG`.
# Substitute BOOST_PYTHON_VARIABLE.
m4_define([_BOOST_PYTHON_CONFIG],
[AC_SUBST([BOOST_PYTHON_$1],
          [`python-config --$2 2>/dev/null`])dnl
boost_python_save_$1=$$1
$1="$$1 $BOOST_PYTHON_$1"])


# BOOST_PYTHON([PREFERRED-RT-OPT])
# --------------------------------
# Look for Boost.Python.  For the documentation of PREFERRED-RT-OPT,
# see the documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Python],
[_BOOST_PYTHON_CONFIG([CPPFLAGS], [includes])
_BOOST_PYTHON_CONFIG([LDFLAGS],   [ldflags])
_BOOST_PYTHON_CONFIG([LIBS],      [libs])
m4_pattern_allow([^BOOST_PYTHON_MODULE$])dnl
BOOST_FIND_LIBS([python], [python python3], [$1],
                [boost/python.hpp],
                [], [BOOST_PYTHON_MODULE(empty) {}])
CPPFLAGS=$boost_python_save_CPPFLAGS
LDFLAGS=$boost_python_save_LDFLAGS
LIBS=$boost_python_save_LIBS
])# BOOST_PYTHON


# BOOST_REF()
# -----------
# Look for Boost.Ref
BOOST_DEFUN([Ref],
[BOOST_FIND_HEADER([boost/ref.hpp])])


# BOOST_REGEX([PREFERRED-RT-OPT])
# -------------------------------
# Look for Boost.Regex.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Regex],
[BOOST_FIND_LIB([regex], [$1],
                [boost/regex.hpp],
                [boost::regex exp("*"); boost::regex_match("foo", exp);])
])# BOOST_REGEX


# BOOST_SERIALIZATION([PREFERRED-RT-OPT])
# ---------------------------------------
# Look for Boost.Serialization.  For the documentation of PREFERRED-RT-OPT, see
# the documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Serialization],
[BOOST_FIND_LIB([serialization], [$1],
                [boost/archive/text_oarchive.hpp],
                [std::ostream* o = 0; // Cheap way to get an ostream...
                boost::archive::text_oarchive t(*o);])
])# BOOST_SERIALIZATION


# BOOST_SIGNALS([PREFERRED-RT-OPT])
# ---------------------------------
# Look for Boost.Signals.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Signals],
[BOOST_FIND_LIB([signals], [$1],
                [boost/signal.hpp],
                [boost::signal<void ()> s;])
])# BOOST_SIGNALS


# BOOST_SIGNALS2()
# ----------------
# Look for Boost.Signals2 (new since 1.39.0).
BOOST_DEFUN([Signals2],
[BOOST_FIND_HEADER([boost/signals2.hpp])
])# BOOST_SIGNALS2


# BOOST_SMART_PTR()
# -----------------
# Look for Boost.SmartPtr
BOOST_DEFUN([Smart_Ptr],
[BOOST_FIND_HEADER([boost/scoped_ptr.hpp])
BOOST_FIND_HEADER([boost/shared_ptr.hpp])
])


# BOOST_STATICASSERT()
# --------------------
# Look for Boost.StaticAssert
BOOST_DEFUN([StaticAssert],
[BOOST_FIND_HEADER([boost/static_assert.hpp])])


# BOOST_STRING_ALGO()
# -------------------
# Look for Boost.StringAlgo
BOOST_DEFUN([String_Algo],
[BOOST_FIND_HEADER([boost/algorithm/string.hpp])
])


# BOOST_SYSTEM([PREFERRED-RT-OPT])
# --------------------------------
# Look for Boost.System.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.  This library was introduced in Boost
# 1.35.0.
BOOST_DEFUN([System],
[BOOST_FIND_LIB([system], [$1],
                [boost/system/error_code.hpp],
                [boost::system::error_code e; e.clear();])
])# BOOST_SYSTEM


# BOOST_TEST([PREFERRED-RT-OPT])
# ------------------------------
# Look for Boost.Test.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Test],
[m4_pattern_allow([^BOOST_CHECK$])dnl
BOOST_FIND_LIB([unit_test_framework], [$1],
               [boost/test/unit_test.hpp], [BOOST_CHECK(2 == 2);],
               [using boost::unit_test::test_suite;
               test_suite* init_unit_test_suite(int argc, char ** argv)
               { return NULL; }])
])# BOOST_TEST


# BOOST_THREAD([PREFERRED-RT-OPT])
# ---------------------------------
# Look for Boost.Thread.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Thread],
[dnl Having the pthread flag is required at least on GCC3 where
dnl boost/thread.hpp would complain if we try to compile without
dnl -pthread on GNU/Linux.
AC_REQUIRE([_BOOST_PTHREAD_FLAG])dnl
boost_thread_save_LIBS=$LIBS
boost_thread_save_LDFLAGS=$LDFLAGS
boost_thread_save_CPPFLAGS=$CPPFLAGS
# Link-time dependency from thread to system was added as of 1.49.0.
if test $boost_major_version -ge 149; then
BOOST_SYSTEM([$1])
fi # end of the Boost.System check.
m4_pattern_allow([^BOOST_SYSTEM_(LIBS|LDFLAGS)$])dnl
LIBS="$LIBS $BOOST_SYSTEM_LIBS $boost_cv_pthread_flag"
LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS"
CPPFLAGS="$CPPFLAGS $boost_cv_pthread_flag"

# When compiling for the Windows platform, the threads library is named
# differently.
case $host_os in
  (*mingw*) boost_thread_lib_ext=_win32;;
esac
BOOST_FIND_LIBS([thread], [thread$boost_thread_lib_ext],
                [$1],
                [boost/thread.hpp], [boost::thread t; boost::mutex m;])

BOOST_THREAD_LIBS="$BOOST_THREAD_LIBS $BOOST_SYSTEM_LIBS $boost_cv_pthread_flag"
BOOST_THREAD_LDFLAGS="$BOOST_SYSTEM_LDFLAGS"
BOOST_CPPFLAGS="$BOOST_CPPFLAGS $boost_cv_pthread_flag"
LIBS=$boost_thread_save_LIBS
LDFLAGS=$boost_thread_save_LDFLAGS
CPPFLAGS=$boost_thread_save_CPPFLAGS
])# BOOST_THREAD

AU_ALIAS([BOOST_THREADS], [BOOST_THREAD])


# BOOST_TOKENIZER()
# -----------------
# Look for Boost.Tokenizer
BOOST_DEFUN([Tokenizer],
[BOOST_FIND_HEADER([boost/tokenizer.hpp])])


# BOOST_TRIBOOL()
# ---------------
# Look for Boost.Tribool
BOOST_DEFUN([Tribool],
[BOOST_FIND_HEADER([boost/logic/tribool_fwd.hpp])
BOOST_FIND_HEADER([boost/logic/tribool.hpp])
])


# BOOST_TUPLE()
# -------------
# Look for Boost.Tuple
BOOST_DEFUN([Tuple],
[BOOST_FIND_HEADER([boost/tuple/tuple.hpp])])


# BOOST_TYPETRAITS()
# --------------------
# Look for Boost.TypeTraits
BOOST_DEFUN([TypeTraits],
[BOOST_FIND_HEADER([boost/type_traits.hpp])])


# BOOST_UTILITY()
# ---------------
# Look for Boost.Utility (noncopyable, result_of, base-from-member idiom,
# etc.)
BOOST_DEFUN([Utility],
[BOOST_FIND_HEADER([boost/utility.hpp])])


# BOOST_VARIANT()
# ---------------
# Look for Boost.Variant.
BOOST_DEFUN([Variant],
[BOOST_FIND_HEADER([boost/variant/variant_fwd.hpp])
BOOST_FIND_HEADER([boost/variant.hpp])])


# BOOST_POINTER_CONTAINER()
# ------------------------
# Look for Boost.PointerContainer
BOOST_DEFUN([Pointer_Container],
[BOOST_FIND_HEADER([boost/ptr_container/ptr_deque.hpp])
BOOST_FIND_HEADER([boost/ptr_container/ptr_list.hpp])
BOOST_FIND_HEADER([boost/ptr_container/ptr_vector.hpp])
BOOST_FIND_HEADER([boost/ptr_container/ptr_array.hpp])
BOOST_FIND_HEADER([boost/ptr_container/ptr_set.hpp])
BOOST_FIND_HEADER([boost/ptr_container/ptr_map.hpp])
])# BOOST_POINTER_CONTAINER


# BOOST_WAVE([PREFERRED-RT-OPT])
# ------------------------------
# NOTE: If you intend to use Wave/Spirit with thread support, make sure you
# call BOOST_THREAD first.
# Look for Boost.Wave.  For the documentation of PREFERRED-RT-OPT, see the
# documentation of BOOST_FIND_LIB above.
BOOST_DEFUN([Wave],
[AC_REQUIRE([BOOST_FILESYSTEM])dnl
AC_REQUIRE([BOOST_DATE_TIME])dnl
boost_wave_save_LIBS=$LIBS
boost_wave_save_LDFLAGS=$LDFLAGS
m4_pattern_allow([^BOOST_((FILE)?SYSTEM|DATE_TIME|THREAD)_(LIBS|LDFLAGS)$])dnl
LIBS="$LIBS $BOOST_SYSTEM_LIBS $BOOST_FILESYSTEM_LIBS $BOOST_DATE_TIME_LIBS \
$BOOST_THREAD_LIBS"
LDFLAGS="$LDFLAGS $BOOST_SYSTEM_LDFLAGS $BOOST_FILESYSTEM_LDFLAGS \
$BOOST_DATE_TIME_LDFLAGS $BOOST_THREAD_LDFLAGS"
BOOST_FIND_LIB([wave], [$1],
                [boost/wave.hpp],
                [boost::wave::token_id id; get_token_name(id);])
LIBS=$boost_wave_save_LIBS
LDFLAGS=$boost_wave_save_LDFLAGS
])# BOOST_WAVE


# BOOST_XPRESSIVE()
# -----------------
# Look for Boost.Xpressive (new since 1.36.0).
BOOST_DEFUN([Xpressive],
[BOOST_FIND_HEADER([boost/xpressive/xpressive.hpp])])


# ----------------- #
# Internal helpers. #
# ----------------- #


# _BOOST_PTHREAD_FLAG()
# ---------------------
# Internal helper for BOOST_THREAD.  Computes boost_cv_pthread_flag
# which must be used in CPPFLAGS and LIBS.
#
# Yes, we *need* to put the -pthread thing in CPPFLAGS because with GCC3,
# boost/thread.hpp will trigger a #error if -pthread isn't used:
#   boost/config/requires_threads.hpp:47:5: #error "Compiler threading support
#   is not turned on. Please set the correct command line options for
#   threading: -pthread (Linux), -pthreads (Solaris) or -mthreads (Mingw32)"
#
# Based on ACX_PTHREAD: http://autoconf-archive.cryp.to/acx_pthread.html
AC_DEFUN([_BOOST_PTHREAD_FLAG],
[AC_REQUIRE([AC_PROG_CXX])dnl
AC_REQUIRE([AC_CANONICAL_HOST])dnl
AC_LANG_PUSH([C++])dnl
AC_CACHE_CHECK([for the flags needed to use pthreads], [boost_cv_pthread_flag],
[ boost_cv_pthread_flag=
  # The ordering *is* (sometimes) important.  Some notes on the
  # individual items follow:
  # (none): in case threads are in libc; should be tried before -Kthread and
  #       other compiler flags to prevent continual compiler warnings
  # -lpthreads: AIX (must check this before -lpthread)
  # -Kthread: Sequent (threads in libc, but -Kthread needed for pthread.h)
  # -kthread: FreeBSD kernel threads (preferred to -pthread since SMP-able)
  # -llthread: LinuxThreads port on FreeBSD (also preferred to -pthread)
  # -pthread: GNU Linux/GCC (kernel threads), BSD/GCC (userland threads)
  # -pthreads: Solaris/GCC
  # -mthreads: MinGW32/GCC, Lynx/GCC
  # -mt: Sun Workshop C (may only link SunOS threads [-lthread], but it
  #      doesn't hurt to check since this sometimes defines pthreads too;
  #      also defines -D_REENTRANT)
  #      ... -mt is also the pthreads flag for HP/aCC
  # -lpthread: GNU Linux, etc.
  # --thread-safe: KAI C++
  case $host_os in #(
    *solaris*)
      # On Solaris (at least, for some versions), libc contains stubbed
      # (non-functional) versions of the pthreads routines, so link-based
      # tests will erroneously succeed.  (We need to link with -pthreads/-mt/
      # -lpthread.)  (The stubs are missing pthread_cleanup_push, or rather
      # a function called by this macro, so we could check for that, but
      # who knows whether they'll stub that too in a future libc.)  So,
      # we'll just look for -pthreads and -lpthread first:
      boost_pthread_flags="-pthreads -lpthread -mt -pthread";; #(
    *)
      boost_pthread_flags="-lpthreads -Kthread -kthread -llthread -pthread \
                           -pthreads -mthreads -lpthread --thread-safe -mt";;
  esac
  # Generate the test file.
  AC_LANG_CONFTEST([AC_LANG_PROGRAM([#include <pthread.h>],
    [pthread_t th; pthread_join(th, 0);
    pthread_attr_init(0); pthread_cleanup_push(0, 0);
    pthread_create(0,0,0,0); pthread_cleanup_pop(0);])])
  for boost_pthread_flag in '' $boost_pthread_flags; do
    boost_pthread_ok=false
dnl Re-use the test file already generated.
    boost_pthreads__save_LIBS=$LIBS
    LIBS="$LIBS $boost_pthread_flag"
    AC_LINK_IFELSE([],
      [if grep ".*$boost_pthread_flag" conftest.err; then
         echo "This flag seems to have triggered warnings" >&AS_MESSAGE_LOG_FD
       else
         boost_pthread_ok=:; boost_cv_pthread_flag=$boost_pthread_flag
       fi])
    LIBS=$boost_pthreads__save_LIBS
    $boost_pthread_ok && break
  done
])
AC_LANG_POP([C++])dnl
])# _BOOST_PTHREAD_FLAG


# _BOOST_gcc_test(MAJOR, MINOR)
# -----------------------------
# Internal helper for _BOOST_FIND_COMPILER_TAG.
m4_define([_BOOST_gcc_test],
["defined __GNUC__ && __GNUC__ == $1 && __GNUC_MINOR__ == $2 && !defined __ICC @ gcc$1$2"])dnl

# _BOOST_mingw_test(MAJOR, MINOR)
# -----------------------------
# Internal helper for _BOOST_FIND_COMPILER_TAG.
m4_define([_BOOST_mingw_test],
["defined __GNUC__ && __GNUC__ == $1 && __GNUC_MINOR__ == $2 && !defined __ICC && \
  (defined WIN32 || defined WINNT || defined _WIN32 || defined __WIN32 \
         || defined __WIN32__ || defined __WINNT || defined __WINNT__) @ mgw$1$2"])dnl


# _BOOST_FIND_COMPILER_TAG()
# --------------------------
# Internal.  When Boost is installed without --layout=system, each library
# filename will hold a suffix that encodes the compiler used during the
# build.  The Boost build system seems to call this a `tag'.
AC_DEFUN([_BOOST_FIND_COMPILER_TAG],
[AC_REQUIRE([AC_PROG_CXX])dnl
AC_REQUIRE([AC_CANONICAL_HOST])dnl
AC_CACHE_CHECK([for the toolset name used by Boost for $CXX],
               [boost_cv_lib_tag],
[boost_cv_lib_tag=unknown
if test x$boost_cv_inc_path != xno; then
  AC_LANG_PUSH([C++])dnl
  # The following tests are mostly inspired by boost/config/auto_link.hpp
  # The list is sorted to most recent/common to oldest compiler (in order
  # to increase the likelihood of finding the right compiler with the
  # least number of compilation attempt).
  # Beware that some tests are sensible to the order (for instance, we must
  # look for MinGW before looking for GCC3).
  # I used one compilation test per compiler with a #error to recognize
  # each compiler so that it works even when cross-compiling (let me know
  # if you know a better approach).
  # Known missing tags (known from Boost's tools/build/v2/tools/common.jam):
  #   como, edg, kcc, bck, mp, sw, tru, xlc
  # I'm not sure about my test for `il' (be careful: Intel's ICC pre-defines
  # the same defines as GCC's).
  for i in \
    _BOOST_mingw_test(5, 0) \
    _BOOST_gcc_test(5, 0) \
    _BOOST_mingw_test(4, 10) \
    _BOOST_gcc_test(4, 10) \
    _BOOST_mingw_test(4, 9) \
    _BOOST_gcc_test(4, 9) \
    _BOOST_mingw_test(4, 8) \
    _BOOST_gcc_test(4, 8) \
    _BOOST_mingw_test(4, 7) \
    _BOOST_gcc_test(4, 7) \
    _BOOST_mingw_test(4, 6) \
    _BOOST_gcc_test(4, 6) \
    _BOOST_mingw_test(4, 5) \
    _BOOST_gcc_test(4, 5) \
    _BOOST_mingw_test(4, 4) \
    _BOOST_gcc_test(4, 4) \
    _BOOST_mingw_test(4, 3) \
    _BOOST_gcc_test(4, 3) \
    _BOOST_mingw_test(4, 2) \
    _BOOST_gcc_test(4, 2) \
    _BOOST_mingw_test(4, 1) \
    _BOOST_gcc_test(4, 1) \
    _BOOST_mingw_test(4, 0) \
    _BOOST_gcc_test(4, 0) \
    "defined __GNUC__ && __GNUC__ == 3 && !defined __ICC \
     && (defined WIN32 || defined WINNT || defined _WIN32 || defined __WIN32 \
         || defined __WIN32__ || defined __WINNT || defined __WINNT__) @ mgw" \
    _BOOST_gcc_test(3, 4) \
    _BOOST_gcc_test(3, 3) \
    "defined _MSC_VER && _MSC_VER >= 1500 @ vc90" \
    "defined _MSC_VER && _MSC_VER == 1400 @ vc80" \
    _BOOST_gcc_test(3, 2) \
    "defined _MSC_VER && _MSC_VER == 1310 @ vc71" \
    _BOOST_gcc_test(3, 1) \
    _BOOST_gcc_test(3, 0) \
    "defined __BORLANDC__ @ bcb" \
    "defined __ICC && (defined __unix || defined __unix__) @ il" \
    "defined __ICL @ iw" \
    "defined _MSC_VER && _MSC_VER == 1300 @ vc7" \
    _BOOST_gcc_test(2, 95) \
    "defined __MWERKS__ && __MWERKS__ <= 0x32FF @ cw9" \
    "defined _MSC_VER && _MSC_VER < 1300 && !defined UNDER_CE @ vc6" \
    "defined _MSC_VER && _MSC_VER < 1300 && defined UNDER_CE @ evc4" \
    "defined __MWERKS__ && __MWERKS__ <= 0x31FF @ cw8"
  do
    boost_tag_test=`expr "X$i" : 'X\([[^@]]*\) @ '`
    boost_tag=`expr "X$i" : 'X[[^@]]* @ \(.*\)'`
    AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
#if $boost_tag_test
/* OK */
#else
# error $boost_tag_test
#endif
]])], [boost_cv_lib_tag=$boost_tag; break], [])
  done
AC_LANG_POP([C++])dnl
  case $boost_cv_lib_tag in #(
    # Some newer (>= 1.35?) versions of Boost seem to only use "gcc" as opposed
    # to "gcc41" for instance.
    *-gcc | *'-gcc ') :;; #(  Don't re-add -gcc: it's already in there.
    gcc*)
      boost_tag_x=
      case $host_os in #(
        darwin*)
          if test $boost_major_version -ge 136; then
            # The `x' added in r46793 of Boost.
            boost_tag_x=x
          fi;;
      esac
      # We can specify multiple tags in this variable because it's used by
      # BOOST_FIND_LIB that does a `for tag in -$boost_cv_lib_tag' ...
      boost_cv_lib_tag="$boost_tag_x$boost_cv_lib_tag -${boost_tag_x}gcc"
      ;; #(
    unknown)
      AC_MSG_WARN([[could not figure out which toolset name to use for $CXX]])
      boost_cv_lib_tag=
      ;;
  esac
fi])dnl end of AC_CACHE_CHECK
])# _BOOST_FIND_COMPILER_TAG


# _BOOST_GUESS_WHETHER_TO_USE_MT()
# --------------------------------
# Compile a small test to try to guess whether we should favor MT (Multi
# Thread) flavors of Boost.  Sets boost_guess_use_mt accordingly.
AC_DEFUN([_BOOST_GUESS_WHETHER_TO_USE_MT],
[# Check whether we do better use `mt' even though we weren't ask to.
AC_LANG_PUSH([C++])dnl
AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
#if defined _REENTRANT || defined _MT || defined __MT__
/* use -mt */
#else
# error MT not needed
#endif
]])], [boost_guess_use_mt=:], [boost_guess_use_mt=false])
AC_LANG_POP([C++])dnl
])

# _BOOST_AC_LINK_IFELSE(PROGRAM, [ACTION-IF-TRUE], [ACTION-IF-FALSE])
# -------------------------------------------------------------------
# Fork of _AC_LINK_IFELSE that preserves conftest.o across calls.  Fragile,
# will break when Autoconf changes its internals.  Requires that you manually
# rm -f conftest.$ac_objext in between to really different tests, otherwise
# you will try to link a conftest.o left behind by a previous test.
# Used to aggressively optimize BOOST_FIND_LIB (see the big comment in this
# macro).
#
# Don't use "break" in the actions, as it would short-circuit some code
# this macro runs after the actions.
m4_define([_BOOST_AC_LINK_IFELSE],
[m4_ifvaln([$1], [AC_LANG_CONFTEST([$1])])dnl
rm -f conftest$ac_exeext
boost_save_ac_ext=$ac_ext
boost_use_source=:
# If we already have a .o, re-use it.  We change $ac_ext so that $ac_link
# tries to link the existing object file instead of compiling from source.
test -f conftest.$ac_objext && ac_ext=$ac_objext && boost_use_source=false &&
  _AS_ECHO_LOG([re-using the existing conftest.$ac_objext])
AS_IF([_AC_DO_STDERR($ac_link) && {
         test -z "$ac_[]_AC_LANG_ABBREV[]_werror_flag" ||
         test ! -s conftest.err
       } && test -s conftest$ac_exeext && {
         test "$cross_compiling" = yes ||
         $as_executable_p conftest$ac_exeext
dnl FIXME: use AS_TEST_X instead when 2.61 is widespread enough.
       }],
      [$2],
      [if $boost_use_source; then
         _AC_MSG_LOG_CONFTEST
       fi
       $3])
ac_objext=$boost_save_ac_objext
ac_ext=$boost_save_ac_ext
dnl Delete also the IPA/IPO (Inter Procedural Analysis/Optimization)
dnl information created by the PGI compiler (conftest_ipa8_conftest.oo),
dnl as it would interfere with the next link command.
rm -f core conftest.err conftest_ipa8_conftest.oo \
      conftest$ac_exeext m4_ifval([$1], [conftest.$ac_ext])[]dnl
])# _BOOST_AC_LINK_IFELSE

# Local Variables:
# mode: autoconf
# End:
#!/bin/bash

start=$(date +%s)
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
cd /vagrant

echo "::::::::::::: COMPILE HEDGEHOG :::::::::::::"
cd /vagrant
./autogen.sh
mkdir build
cd build
../configure
make

echo "::::::::::::: INSTALL HEDGEHOG :::::::::::::"
make install
make install-rpg


echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"

cd /vagrant

end=$(date +%s)

diff=$(( $end - $start ))


echo ":::::::::::::::::::::::::::::::::"
echo "::::: COMPILATION: $diff s"
echo ":::::::::::::::::::::::::::::::::"
#!/bin/bash

start=$(date +%s)
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
cd /vagrant

echo "::::::::::::: REPO :::::::::::::"
add-apt-repository ppa:opencpu/rapache
apt-get update

echo "::::::::::::: C++ :::::::::::::"
apt-get install -y libboost-all-dev libtool libpqxx-dev

echo "::::::::::::: R :::::::::::::"
apt-get install -y r-base r-base-core r-base-dev libcairo2-dev libxt-dev r-cran-ggplot2 r-cran-dbi r-cran-cairodevice r-cran-reshape r-cran-digest
echo "install.packages(c('brew','Cairo','googleVis','RPostgreSQL','R.utils','yaml','dplyr'), repos='http://cran.rstudio.com/')" | R --vanilla

echo "::::::::::::: POSTGRES :::::::::::::"
apt-get install -y postgresql postgresql-client postgresql-server-dev-9.3 postgresql-contrib pgbouncer

echo "::::::::::::: Apache, etc. :::::::::::::"
apt-get install -y apache2-mpm-prefork apache2-prefork-dev git libapache2-mod-r-base libyaml-tiny-perl

echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"

cd /vagrant

end=$(date +%s)

diff=$(( $end - $start ))

echo ":::::::::::::::::::::::::::::::::"
echo "::::: DEPENDENCIES: $diff s"
echo ":::::::::::::::::::::::::::::::::"
#!/bin/bash

start=$(date +%s)
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
cd /vagrant

echo "::::::::::::: CREATE USERS :::::::::::::"
# create a system user and group called hedgehog
addgroup --system hedgehog
adduser --system --ingroup hedgehog hedgehog

chown -R hedgehog:hedgehog /usr/local/var/hedgehog/data
chown -R www-data:www-data /usr/local/var/hedgehog/www

echo "::::::::::::: CONFIGURE APACHE :::::::::::::"
a2dissite 000-default.conf

cp /usr/local/share/hedgehog/conf/hedgehog.conf /etc/apache2/sites-available/
a2ensite hedgehog.conf

echo "
<Directory /usr/local/share/hedgehog>
AllowOverride None
Require all granted
</Directory>

<Directory /usr/local/var/hedgehog/www>
AllowOverride None
Require all granted
</Directory>" >> /etc/apache2/apache2.conf

#apache/rapache write some of their logs to user.* so it can be useful to change the syslog config: Uncomment the line beginning 'user.*'.
#sudo vi /etc/rsyslog.d/50-default.conf

service apache2 restart

#Edit the the <prefix>/etc/hedgehog/nodes.csv to specify the servers, nodes and grouping to be used (example format is provided with entries commented out).
#sudo -u postgres /usr/local/bin/database_create
#sudo -u hedgehog /usr/local/bin/hedgehogctl database_init
echo "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<"

cd /vagrant

end=$(date +%s)

diff=$(( $end - $start ))

echo ":::::::::::::::::::::::::::::::::"
echo "::::: CONFIGURATION: $diff s"
echo ":::::::::::::::::::::::::::::::::"
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/**
 */

#define BOOST_FILESYSTEM_NO_DEPRECATED

#include <string>
#include <map>
#include <exception>
#include <iostream>
#include <time.h>

#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/xml_parser.hpp>
#include <boost/foreach.hpp>
#include <boost/algorithm/string.hpp>
#include <boost/filesystem.hpp>

#include "pqxx/pqxx"

#include "dsc_types.h"
#include "DSCDataManager.h"
#include "DSCStrategy.h"
#include "DSCStrategyFactory.h"

#include "Format1DSCStrategy.h"
#include "Format2DSCStrategy.h"
#include "Format3aDSCStrategy.h"
#include "Format3bDSCStrategy.h"
#include "Format4DSCStrategy.h"

using namespace std;

DSCDataManager::DSCDataManager(const string& server, const string& node, pqxx::work *pg_db_trans) {

    // Use the transaction pointer to decide if the output will be a database
    if ( pg_db_trans == NULL ) {
        using_pg_db_ = false;
        pg_db_transaction_ = NULL;
        server_ = server;
        node_ = node;
    } else {
        using_pg_db_ = true;
        pg_db_transaction_ = pg_db_trans;
        server_ = server;
        node_ = node;
        server_id_ = get_server_id(server, *pg_db_trans);
        node_id_ = get_node_id(server_id_, node, *pg_db_trans);
    }

}

void
DSCDataManager::clear() {

    start_time_ = 0;
    name_.clear();
    time_strings_[0].clear();
    time_strings_[1].clear();
    time_strings_[2].clear();
    counts_.clear();
    preprocess_required_ = false;
    for(vector<DSCStrategy*>::iterator strategy_it = strategies_.begin(); strategy_it != strategies_.end(); ++strategy_it){
        delete *strategy_it;
    }
    strategies_.clear();

}

int
DSCDataManager::load(const string dat_line, vector<DSCStrategy*>::iterator strategy_it) {

    clear();

    // Call out to the strategy factory as the strategies 
    // know how to convert the line of data to a DSCUMap
    if ( using_pg_db_ == true ){
        if ( (*strategy_it)->process_dat_line(counts_, time_strings_, dat_line) != 0 ) return 1;
        struct tm *tmp_time;
        time_t tmp_time1;
        tmp_time1 = (time_t)atoi(time_strings_[1].c_str());
        tmp_time = gmtime(const_cast<const time_t *>(&tmp_time1));
        char yyyymmdd_c[9];
        char yyyy_mm_c[8];
        strftime(yyyymmdd_c, 9, "%Y%m%d", tmp_time);
        time_strings_[0].assign(yyyymmdd_c,8);
        strftime(yyyy_mm_c, 8, "%Y_%m", tmp_time);
        time_strings_[2].assign(yyyy_mm_c,7);
    } else {
        cerr << "Error: Attempt to write a dat file to a dat file for graph " << name_ << endl;
        exit(1);
    }
    return 0;
    
}

int
DSCDataManager::load(const boost::filesystem::path& file_path, vector<DSCStrategy*>::iterator strategy_it) {

    clear();

    name_ = file_path.filename().string();
    
    struct tm tm;
    std::stringstream ss;
    string tmp_time_string;
    char yyyy_mm_c[8];

    //  The only time/date we know of is that contained in the path 
    //  to the dat file.
    time_strings_[0].assign(file_path.parent_path().filename().string());
    tmp_time_string = time_strings_[0] + " 00:00:00";
    if ( strptime(tmp_time_string.c_str(), "%Y%m%d %H:%M:%S", &tm) != NULL ) {
        ss << mktime(&tm);
        time_strings_[1] = ss.str();
    strftime(yyyy_mm_c, 8, "%Y_%m", &tm);
    time_strings_[2].assign(yyyy_mm_c,7);

    } else {
        cerr << "Error: Unable to extract unixtime when loading dat file " << file_path.string() << endl;
        return 1;
    }

    // Call out to the strategy factory as the strategies 
    // know how to convert the file to a DSCUMap
    if ( using_pg_db_ == true ){
        (*strategy_it)->process_dat_file(counts_, time_strings_, file_path);
    } else {
        cerr << "Error: Attempt to write a dat file to a dat file for graph " << name_ << endl;
        exit(1);
    }
    return 0;

}

int 
DSCDataManager::load(const boost::property_tree::ptree::value_type &array_value, bool rssac) {

    int dim_counter = 0; 
    string dim_type[2];
    string dim1_value;
    string dim2_value;

    dim_type[0].clear();
    dim_type[1].clear();
    dim1_value.clear();
    dim2_value.clear();
    
    clear();
    preprocess_required_ = true;

    // catch any generic XML processing errors
    try {
	    if ( boost::equals ( array_value.first, "array") ) {
	        BOOST_FOREACH(boost::property_tree::ptree::value_type const& value, array_value.second) {
	            // Get attributes from the array element
	            if ( boost::equals ( value.first, "<xmlattr>") ) {
	                name_       = value.second.get<string>("name");
	                start_time_ = value.second.get<int>("start_time");

	                // round start time down to start of the minute
	                start_time_ = int(start_time_ / 60) * 60;

	                struct tm *tmp_time;
	                time_t tmp_time1;
	                tmp_time1 = (time_t)start_time_;
	                tmp_time = gmtime(const_cast<const time_t *>(&tmp_time1));
	                char yyyymmdd_c[9];
	                char yyyy_mm_c[8];
	                strftime(yyyymmdd_c, 9, "%Y%m%d", tmp_time);
	                time_strings_[0].assign(yyyymmdd_c,8);
	                // don't use strftime here due to timezone issues
	                ostringstream tmp_time2;
	                tmp_time2 << start_time_;
	                time_strings_[1] = tmp_time2.str();
	                strftime(yyyy_mm_c, 8, "%Y_%m", tmp_time);
	                time_strings_[2].assign(yyyy_mm_c,7);
	            }

	            // Get dimensions (Should always be 2 of them)
	            if ( boost::equals ( value.first, "dimension") ) {
	                BOOST_FOREACH(boost::property_tree::ptree::value_type const& dim_value, value.second) {
	                    if ( boost::equals ( dim_value.first, "<xmlattr>") ) {
	                        dim_type[dim_counter] = dim_value.second.get<string>("type");
	                        dim_counter++;
	                    }
	                }
	            }

	            // Get data
	            if ( boost::equals ( value.first, "data") ) {
	                BOOST_FOREACH(boost::property_tree::ptree::value_type const& value1, value.second) {
	                    if ( boost::equals ( value1.first, dim_type[0]) ) {
	                        BOOST_FOREACH(boost::property_tree::ptree::value_type const& value2, value1.second) {
	                            if ( boost::equals ( value2.first, "<xmlattr>") ) {
	                                dim1_value = value2.second.get<string>("val");
	                            } else if ( boost::equals ( value2.first, dim_type[1]) ) {
	                                int tmp_count = 0;
	                                BOOST_FOREACH(boost::property_tree::ptree::value_type const& value3, value2.second) {
	                                    if ( boost::equals ( value3.first, "<xmlattr>") ) {
	                                        dim2_value = value3.second.get<string>("val");
	                                        tmp_count  = value3.second.get<int>("count");
	                                    }
	                                }
	                                DSCKey tmp_key(dim1_value,dim2_value);
	                                counts_.insert(pair<DSCKey,int>(tmp_key, tmp_count));
	                            }
	                        }
	                    }
	                }
	            }
	        }
	    }
	}
    catch( std::exception & e )
    {
        cerr << "Exception loading XML file: " << e.what() << endl;
        return 1;
    }
    catch( ... )
    {
        cerr << "Unknown exception caught" << endl;
        return 1;
    }

  strategies_ = DSCStrategyFactory::createStrategy(server_, name_, rssac);
  return 0;

}

int
DSCDataManager::process() {

    int status = 0;

    // if no data, don't process
    if(counts_.size() == 0)
        return 0;
    
    long unsigned int i=0;
    vector<DSCStrategy*>::iterator strategy_it;
    
    for(strategy_it = strategies_.begin(); strategy_it != strategies_.end(); strategy_it++){
        DSCUMap *output_counts = &counts_;
        DSCUMap new_counts;
        // preprocess is only required if the input channel was XML
        if (preprocess_required_) {
           (*strategy_it)->preprocess_data(counts_, new_counts, time_strings_, server_id_, node_id_, pg_db_transaction_);
           output_counts = &new_counts;
        }
        if ( using_pg_db_ == true ) {
            status = (*strategy_it)->process_to_db(*output_counts, time_strings_, server_id_, node_id_, *pg_db_transaction_);
            if (status) return 1;
        }
        if ( using_pg_db_ == false ) {
            (*strategy_it)->process_to_dat(*output_counts, time_strings_);
        }
        i++;
    }
    return 0;

}

int
DSCDataManager::process(vector<DSCStrategy*>::iterator strategy_it) {

    int status=0;
    // if no data, don't process
    if(counts_.size() == 0)
        return 0;

    DSCUMap *output_counts = &counts_;
    DSCUMap new_counts;
    if (preprocess_required_){
       (*strategy_it)->preprocess_data(counts_, new_counts, time_strings_, server_id_, node_id_, pg_db_transaction_);
       output_counts = &new_counts;
    }
    if ( using_pg_db_ == true ) {
        status = (*strategy_it)->process_to_db(*output_counts, time_strings_, server_id_, node_id_, *pg_db_transaction_);
        if (status) return 1;
    } else {
        cerr << "Error: Expected to use a database but no transaction found." << endl;
        exit(1);
    }
    return 0;

}

int
DSCDataManager::get_server_id(const string& server, pqxx::work& pg_db_trans) {

    stringstream sql;
    try {
        sql.clear();
        sql.str("");
        sql << "SELECT id FROM dsc.server WHERE name = '" << server << "'" << endl;
        pqxx::result r = pg_db_trans.exec(sql.str());
        if ( r.size() != 1 ) {
            cerr << "Error: Expected 1 server with name " << server << ", "
                 << "but found " << r.size() << endl;
            exit( EXIT_FAILURE );
        }
        return r[0][0].as<int>();
    }
    catch( runtime_error & e )
    {
        cerr << "Runtime error: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( std::exception & e )
    {
        cerr << "Exception: " << e.what() << sql.str() << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }

}

int
DSCDataManager::get_node_id(int server_id, const string& node, pqxx::work& pg_db_trans) {

    stringstream sql;
    try {
        sql.clear();
        sql.str("");
        sql << "SELECT id FROM dsc.node WHERE name = '" << node << "' AND server_id = '" << server_id << "'" << endl;
        pqxx::result r = pg_db_trans.exec(sql.str());
        if ( r.size() != 1 ) {
            cerr << "Error: Expected 1 node with name " << node << " and server ID " << server_id << ", "
                 << "but found " << r.size() << endl;
            exit( EXIT_FAILURE );
        }
        return r[0][0].as<int>();
    }
    catch( runtime_error & e )
    {
        cerr << "Runtime error: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( std::exception & e )
    {
        cerr << "Exception: " << e.what() << sql.str() << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }

}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#include <string>
#include <map>
#include <time.h>

#include <boost/property_tree/ptree.hpp>
#include <boost/unordered_map.hpp>

#include "pqxx/pqxx"

#include "dsc_types.h"

using namespace std;

class DSCStrategy;


/// \brief Class that parses the DSC data and calls an appropriate strategy to 
/// to deal with the data.
/// 
/// This class is responsible for owning the data and providing load functions 
/// for the various input channels and provides functions to process the data 
/// using an appropriate strategy.
class DSCDataManager
{
    
private:
    
    // ****************
    // Member data
    // ****************
    /// \brief string to hold the name of the array
    std::string name_;

    /// \brief integer start time read from the xml
    int start_time_;

    /// \brief array of strings to hold time is a variety of formats
    std::string time_strings_[3];
    /// \brief unordered map (defined in dsc_types.h) containing the data
    /// loaded from the xml
    DSCUMap counts_;
    /// \brief A vetor holding the strategies returned by the factory
    std::vector<DSCStrategy*> strategies_;
    /// \brief Boolean to tell if the output is a database or a dat file
    bool using_pg_db_;
    /// \brief Pointer to a PostgreSQL transaction.
    pqxx::work *pg_db_transaction_;
    /// \brief The name of the server node.
    std::string node_;
    /// \brief The name of the server.
    std::string server_;
    
    int node_id_;
    int server_id_;
    bool preprocess_required_;
    int get_server_id(const string& server, pqxx::work& pg_db_trans);
    int get_node_id(int server_id, const string& node, pqxx::work& pg_db_trans);
	
public:
    
    // ****************
    // Constructor
    // ****************
    /// \brief Initialize the class making use of the pointer to the database 
    /// transaction to determine the output destination.
    //  TODO: allow configure to decide if db functionality is compiled in
    DSCDataManager(const string& server, const string& node, pqxx::work *pg_db_trans);
            
    /// \brief Method to clear the member data. This class is only constructed 
    /// once and repeatedly reused by the DSCIOManager.
    void clear();
    
    /// \brief Method to parse the xml held in a boost::property_tree::ptree
    ///        and populate the member data.
    int load(const boost::property_tree::ptree::value_type &array_value, bool rssac);
    /// \brief Alternate load method that can parse an entire dat file into the 
    /// member data.
    int load(const boost::filesystem::path&, vector<DSCStrategy*>::iterator strategy_it);
    /// \brief Alternate load method that can parse a single line from a dat 
    /// file into the member data.
    int load(const std::string dat_line, vector<DSCStrategy*>::iterator strategy_it);
    /// \brief Process method that is used when processing xml input via  
    /// multiple strategies.
    int process();
    /// \brief Alternate process method that uses a specific strategy to process
    /// the data.
    int process(vector<DSCStrategy*>::iterator strategy_it);
    
};
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   xml_files.cpp
 */

#define BOOST_FILESYSTEM_NO_DEPRECATED

#include <exception>
#include <iostream>
#include <algorithm>
#include <vector>
#include <exception>
#include <time.h>

#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/xml_parser.hpp>
#include <boost/foreach.hpp>
#include <boost/filesystem.hpp>

#include <pqxx/pqxx>

#include "DSCIOManager.h"
#include "DSCDataManager.h"
#include "dsc_types.h"
#include "DSCStrategyFactory.h"

#include "config.h"

using namespace std;
namespace bfs = boost::filesystem;

// Helper function
// Get current date/time, format is YYYY-MM-DD HH:mm:ss time-zone
const std::string currentDateTime() {

    time_t     now = time(0);
    struct tm  tstruct;
    char       buf[80];
    tstruct = *localtime(&now);
    strftime(buf, sizeof(buf), "%Y-%m-%d %X %Z", &tstruct);

    return buf;

}

void replace_string(std::string& str, const std::string& from, const std::string& to) {
    if(from.empty())
        return;
    size_t start_pos = 0;
    while((start_pos = str.find(from, start_pos)) != std::string::npos) {
        str.replace(start_pos, from.length(), to);
        start_pos += to.length(); // In case 'to' contains 'from', like replacing 'x' with 'yx'
    }
}


DSCIOManager::DSCIOManager(const enum INPUT_OUTPUT input, const enum INPUT_OUTPUT output, const string& conn_string, const string& processing_start_date_string, bool rssac) {

    input_ = input;
    output_ = output;
    conn_string_ = conn_string;
    pg_db_conn_ = NULL;
    pg_db_trans_ = NULL;
    processing_start_date_ = NULL;
    rssac_=rssac;

    cout << endl << endl << "----------------------- " << currentDateTime() << " ----------------------" << endl;
    // work out if we have a valid start date
    struct tm p_start_date  = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, NULL };
    if ( ! processing_start_date_string.empty() ) {
       cout << "Processing start date defined as " << processing_start_date_string << endl; 
       if ( strptime(processing_start_date_string.c_str(), "%Y-%m-%d", &p_start_date) != NULL ) {   
                processing_start_date_ = new time_t(mktime(&p_start_date));
                //cout << "Time is: " << p_start_date.tm_year << " " << p_start_date.tm_mon << " " << p_start_date.tm_mday << " " << p_start_date.tm_hour << " " << p_start_date.tm_min << " " << p_start_date.tm_sec << endl;   
       } else {
           cerr << "Error: Invalid processing start date \'" << processing_start_date_string << "\' provided. Must be YYYY-MM-DD format. Exiting." << endl;
           exit(1);
       }
    } 
}

DSCIOManager::~DSCIOManager() {
    
    cout << "----------------------- " << currentDateTime() << " -----------------------" << endl << endl;
    if ( processing_start_date_ != NULL ) {
        delete processing_start_date_;
    }
}
    
int
DSCIOManager::dsc_connect_pg_db() {
    pg_db_conn_ = new pqxx::connection( conn_string_ );
    if ( pg_db_conn_ == NULL ) {
        cerr << "Error connecting to database." << endl;
        exit(1);
    }
    pg_db_conn_->activate();
    
    /* Check Database version */
    int db_version = 0;
    try {
        pg_db_trans_ = new pqxx::work( *pg_db_conn_ );
        if ( pg_db_trans_ == NULL ) {
            cerr << "Error starting transaction." << endl;
            exit(1);
        }
        const pqxx::result R = pg_db_trans_->exec("select version from version;");
        pqxx::result::const_iterator row;
        for (row = R.begin(); row != R.end(); ++row) {
            row["version"].to(db_version);
        }
        if ( db_version != DB_SCHEMA_VERSION ) {
            cout << "Database version is not supported. Expected: " << DB_SCHEMA_VERSION << " Found: " << db_version << endl;
            exit( EXIT_FAILURE);
        }
        if ( pg_db_trans_ != NULL ) {
            pg_db_trans_->commit();
            delete pg_db_trans_;
            pg_db_trans_ = NULL;
        }
    }
    catch ( pqxx::unique_violation & e )
    {
        cerr << "Unique_violation error: " << e.what() << endl;
        exit( EXIT_FAILURE);
    }
    catch( runtime_error & e )
    {
        cerr << "Runtime error: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( std::exception & e )
    {
        cerr << "Exception: " << e.what() << " select version from version;" << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }
    
    return 0;

}

int
DSCIOManager::dsc_disconnect_pg_db() {

    pg_db_conn_->disconnect();
    delete pg_db_conn_;
    // TODO Errors
    return 0;

}

void
DSCIOManager::dsc_activate_ouput_destination() {

    switch (output_) {
        case DAT:
            // Nothing to do
            break;
        case PG_DB:
            dsc_connect_pg_db();
            cout << "*** Database connected" << endl;
            break;
        default:
            // Should never get here!
            cerr << "Error: Unknown output destination." << endl;
            exit(1);
            break;
    }
}

void
DSCIOManager::dsc_deactivate_ouput_destination() {
    switch (output_) {
        case DAT:
            // Nothing to do
            break;
        case PG_DB:
            dsc_disconnect_pg_db();
            cout << "*** Database disconnected" << endl;
            break;
        default:
            // Should never get here!
            cerr << "Error: Unknown output destination." << endl;
            exit(1);
            break;
    }

}

void
DSCIOManager::dsc_import_input_from_source() {

    dsc_populate_files_vector();
    int file_counter=file_vector_.size();
    int file_failures = 0;
    int file_commit_status = 0;  // status of transaction commit
    int data_process_status = 0; // status of data processing
    
    string node = bfs::initial_path().filename().generic_string();
    string server = bfs::initial_path().parent_path().filename().generic_string();
    //fix potential sql injection
    replace(node.begin(), node.end(), '\'', '_');
    replace(server.begin(), server.end(), '\'', '_');
    //transform to 'internal' name
    replace(server.begin(), server.end(), '-', '_');
    replace_string(server, ".", "__");
    transform(server.begin(), server.end(), server.begin(), ::tolower);
    
    cout << "*** Processing server: " << server << endl;
    cout << "*** Processing node:   " << node << endl;
    cout << "*** Found:             " << file_counter << " " <<  file_extension_ << " files to process." << endl;
    
    for (std::vector<bfs::path>::const_iterator file_vector_it (file_vector_.begin()); file_vector_it != file_vector_.end(); ++file_vector_it) {
        if ( pg_db_conn_ != NULL ) {
            pg_db_trans_ = new pqxx::work( *pg_db_conn_ );
            if ( pg_db_trans_ == NULL ) {
                cerr << "Error starting transaction." << endl;
                exit(1);
            }
        }
        cout << endl << currentDateTime() << ": Starting:  " << (*file_vector_it).string() << endl;
        DSCDataManager* dsc_data_manager = new DSCDataManager(server, node, pg_db_trans_);
        file_commit_status = 0;
        
        switch (input_) {
            case XML: {
                // Partially read the xml. This is done using a policy tree,
                // which uses rapidxml to do the parsing. There is no validation
                // performed. Each array element is a "data unit" to send to the
                // data manager load function.
                dsc_xml_ = new boost::property_tree::ptree();
                // If reading fails, exception is thrown.
                try {
                  boost::property_tree::read_xml((*file_vector_it).string(), *dsc_xml_);
                  // We can't validate the XML but make this call the check that the file 
                  // at least has the correct top level node
                  try {
                    dsc_xml_->get_child("dscdata");
                  }
                  catch( std::exception & e ) {
                    cerr << "Error: XML file has no content: " << e.what() << endl;
                    data_process_status = 1;
                    break;					
                  }
                }
                catch( std::exception & e )
                {
                    cerr << "Exception reading XML file: " << e.what() << endl;
                    data_process_status = 1;
                    break;
                }
                catch( ... )
                {
                    cerr << "Unknown exception caught" << endl;
                    data_process_status = 1;
                    break;
                }
                // Get each array element and process it
                BOOST_FOREACH(boost::property_tree::ptree::value_type &array_element, dsc_xml_->get_child("dscdata")) {
                    data_process_status = dsc_data_manager->load(array_element, rssac_);
                    if (data_process_status) break;
                    data_process_status = dsc_data_manager->process();
                    if (data_process_status) break;
                }
                delete dsc_xml_;
            }
            break;
            case DAT: {
                // Call out to strategies to find out how this dat file 
                // needs to be processed.
                // F1, F2 and F3a dat files contain multiple "data units"
                vector<DSCStrategy*>::iterator strategy_it;
                string name = (*file_vector_it).filename().string();
                std::vector<DSCStrategy*> strategies_vector = DSCStrategyFactory::createStrategyDat(server, name);
                bool multi;
                
                if (strategies_vector.size() != 1 ) { 
                    cerr << "Error: Failed to find a single strategy for loading dat file " << (*file_vector_it).string() << " (found: " << strategies_vector.size() << ")" << endl;
                    // exit(1);
                    break;
                }
                
                strategy_it = strategies_vector.begin();
                multi = (*strategy_it)->is_dat_file_multi_unit();

                if ( multi ) {
                    // read the file one line at a time
                    bfs::fstream dat_file;
                    string dat_line;
                    dat_file.open((*file_vector_it), ios_base::in);
                    while (getline(dat_file, dat_line)) {
                        if ( dsc_data_manager->load(dat_line, strategy_it) !=0 ) continue;
                        data_process_status = dsc_data_manager->process(strategy_it);
                        if (data_process_status) break;
                    }
                } else {
                    // The entire file is one "data unit"
                    dsc_data_manager->load((*file_vector_it), strategy_it);
                    data_process_status = dsc_data_manager->process(strategy_it);
                    if (data_process_status) break;
                }
            }
            break;
            default:
                // Should never get here!
                cerr << "Error: Unknown input source" << endl;
                exit(1);
        }
        
        if ( pg_db_trans_ != NULL ) {  
                if ( data_process_status == 0 ) {
                    try {
                        pg_db_trans_->commit();
                    }             
                    catch( std::exception & e ) {
                        cerr << "Commit failed with exception message: " << e.what() << endl;    
                        file_commit_status = 1;             
                    }
                    catch( ... ) {
                        cerr << "Commit failed with general error" << endl;
                        file_commit_status = 1;
                    }
                }
                // Tidy up. If the processing was not successful the transaction will implicitly rollback
                delete pg_db_trans_;
                pg_db_trans_ = NULL;         
        }      
        
        delete dsc_data_manager;        
        
        // move the file to the done directory if all is well
        if (file_commit_status == 0 && data_process_status == 0 ) {
             dsc_move_file_to_done((*file_vector_it));        
        } else {
              dsc_handle_failed_file((*file_vector_it), data_process_status);
            file_failures++;
        }
                
        // Display helpful message. Flush to ensure that it is seen.
        file_counter--;
        cout << currentDateTime() << ": Completed: " << setw(50) << left << (*file_vector_it).string() << "\t" << file_counter << " remaining (" << file_failures << " failures)."<< endl;

        std::flush(cout);
    }
    
    // clear the file vector
    file_vector_.clear();
    
    // endl since the previous cout used \r
    cout << endl;

}

void
DSCIOManager::dsc_populate_files_vector() {
    
    std::string start_of_search_path;
    switch (input_) {
        case XML: 
            start_of_search_path = "./incoming";
            file_extension_ = ".xml";
            break;
        case DAT:
            start_of_search_path = ".";
            file_extension_ = ".dat";
            break;
        default:
            // Should never get here!
            cerr << "Error: Unknown input source" << endl;
            exit(1);
            break;
    }
    
    bfs::path p_start (start_of_search_path);
    bfs::path::iterator path_it;

    // iterators for iterating over all the paths/files in the data directory
    bfs::recursive_directory_iterator it(p_start), end_it;
    try {
        if (bfs::exists(p_start)) {
            if (bfs::is_directory(p_start)) {
                BOOST_FOREACH( bfs::path const &p_search, make_pair(it, end_it) ) {                 
                    if ( p_search.string().find("done") == string::npos ) {
                        if ( ! bfs::is_directory(p_search) ) {
                            if ( p_search.extension() == file_extension_ ) {
                                if (! exclude_file(p_search, start_of_search_path) ) {
                                        file_vector_.push_back(p_search);                                   
                                }
                            }
                        }
                    }
                }
                sort(file_vector_.begin(), file_vector_.end());
            } else {
                cout << p_start << " exists, but is not a directory\n";
                exit(1);
            }
        } else {
            cout << p_start << " does not exist\n";
            exit(1);
        }
    }
    catch (const bfs::filesystem_error& ex) {
        cout << "FS Error: " << ex.what() << endl;
        exit(1);
    }
    catch (std::exception &e) {
        cerr << "Error creating file vector: " << e.what() << endl;
        exit(1);
    }
    
}

void
DSCIOManager::dsc_move_file_to_done(bfs::path path_src) {
    
    bfs::path path_done ("./done");
    std::string done_sub_dir;
    bfs::path path_date;
    bfs::path path_done_date;
    bfs::path path_newfile;
    
    try {
        switch (input_) {
            case XML: 
                done_sub_dir = "dscdata";
                path_date = path_src.parent_path().stem(); //./incoming/yyyymmdd/*.xml --> yyyymmdd
                break;
            case DAT:
                done_sub_dir = "dat";
                path_date = path_src.parent_path().stem(); // ./yyyymmdd/*.dat --> yyyymmdd
                break;
            default:
                // Should never get here!
                cerr << "Error: Unknown input source" << endl;
                exit(1);
                break;
        }
        path_done_date = path_done/path_date/done_sub_dir;
        if (!bfs::exists(path_done_date)) {
                bfs::create_directories(path_done_date);
        }
        path_newfile = path_done_date/path_src.filename();
        bfs::rename(path_src, path_newfile);
        cout << currentDateTime() << ": Moved:     " << path_src.string() << " to " <<  path_newfile.string() << endl;
    }
    catch (const bfs::filesystem_error& ex) {
        cout << "FS Error: " << ex.what() << endl;
        exit(1);
    }
    catch (std::exception &e) {
        cerr << "Error moving file to done: " << e.what() << endl;
        exit(1);
    }
 
}

void
DSCIOManager::dsc_handle_failed_file(bfs::path path_src, int process) {
    
    bfs::path path_newfile;  
    std::string failed = ".failed.";
    std::string reason_string = process ? " Failed to process data " : "Failed to commit data to DB";
    
    try {
        path_newfile = path_src.string() + failed + bfs::unique_path().string();
        bfs::rename(path_src, path_newfile);
        cout << currentDateTime() << ": Error: failed to process: " << path_src.string() << "Reason: " << reason_string << endl;
        cout << currentDateTime() << ": File renamed to:          " << path_newfile.string() << endl;
    }
    catch (const bfs::filesystem_error& ex) {
        cout << "FS Error: " << ex.what() << endl;
        exit(1);
    }
    catch (std::exception &e) {
        cerr << "Error handling failed file: " << e.what() << endl;
        exit(1);
    }
 
}

bool 
DSCIOManager::exclude_file(bfs::path path_src, const string& start_of_search_path) {

    if ( input_ != XML || processing_start_date_ == NULL ) {
        return false;
    }

    // Only do this test on files under './incoming'
    if ( path_src.parent_path().parent_path().string().compare(start_of_search_path) == 0 ) {
        struct tm tm  = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, NULL };
        if ( strptime(path_src.parent_path().filename().string().c_str(), "%Y-%m-%d", &tm) != NULL ) {
            time_t dir_time = mktime(&tm);
            //cout << "Diff time is " << difftime(*processing_start_date_, dir_time) << endl ;
            if ( difftime(*processing_start_date_, dir_time) > 0 )  {
                cout << "File " << path_src.string() << " excluded from processing as it is earlier than the processing start date" << endl;
                return true;
            } else {
                return false;
            }
        }                     
    }
    return false;

}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#ifndef DSCIOMANAGER_H
#define	DSCIOMANAGER_H

#define BOOST_FILESYSTEM_NO_DEPRECATED

#include <vector>

#include <boost/property_tree/ptree.hpp>
#include <boost/filesystem.hpp>

#include <pqxx/pqxx>

#include "dsc_types.h"

namespace bfs = boost::filesystem;

/// \brief DSC IO Manager class.
/// 
/// This class is responsible for configuring input and output channels.
/// Provides public functions to activate and deactivate the output destination
/// and to import the data from the source. It only does minimal parsing of any 
/// input data in order to determine how to proceed. The loading of the data 
/// is left to the data manager.
class DSCIOManager
{
private:
    
    // ****************
    // Member data
    // ****************	
    enum INPUT_OUTPUT input_;
    enum INPUT_OUTPUT output_;
    std::string conn_string_;
    std::string file_extension_;
    std::vector<bfs::path> file_vector_;
    pqxx::connection *pg_db_conn_;
    pqxx::work *pg_db_trans_;
    time_t *processing_start_date_;
    bool rssac_;
    // This is not the intended use of a property tree but it works for me
    boost::property_tree::ptree *dsc_xml_;
    
    /// Populate a vector of filenames.
    void dsc_populate_files_vector();
    /// Move data files that have been successfully processed into a done dir.
    void dsc_move_file_to_done(bfs::path path_src);
    /// Decide what to do with a failed file
    void dsc_handle_failed_file(bfs::path path_src, int process);
    ///  Function to test if a file should be excluded from processing
    bool exclude_file(bfs::path path_src, const string& start_of_search_path);
    /// Connect to PostgreSQL database.
    int dsc_connect_pg_db();
    /// Disconnect from PostgreSQL database.
    int dsc_disconnect_pg_db();


public:
    
    // ****************
    // Constructor
    // ****************
    /// \brief Initialize the class. Sets some of the member variables.
    DSCIOManager(const enum INPUT_OUTPUT input, const enum INPUT_OUTPUT output, const string& conn_string, const string& processing_start_date_string, bool rssac);
    virtual ~DSCIOManager();
    /// \brief Activates the output destination (at the moment this only does 
    /// something for the database case).
    void dsc_activate_ouput_destination();
    /// \brief Deactivates the output destination (at the moment this only does 
    /// something for the database case).
    void dsc_deactivate_ouput_destination();
    /// \brief Import data from either XML or DAT files and calls the data 
    /// manager to process and output the data.
    void dsc_import_input_from_source();
};

#endif
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#include <iostream>
#include <string>
#include <set>
#include <map>

#define BOOST_FILESYSTEM_NO_DEPRECATED
#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/unordered_map.hpp>
#include <boost/lexical_cast.hpp>

#include "dsc_types.h"
#include "DSCStrategy.h"

#include "pqxx/pqxx"

using namespace std;
namespace bfs = boost::filesystem;

DSCStrategy::DSCStrategy(string server, string name, string keys[], int keylength) {

    if ( keylength != 0 ) keys_.insert (keys, keys+keylength);
    server_name_ = server;
    plot_name_ = name;

}

bool
DSCStrategy::is_valid_time(const string& time_string) {

    bool res = true;
    try {
        boost::lexical_cast<time_t>(time_string);
    if (time_string.size() != 10 ) res = false;
    }
    catch (boost::bad_lexical_cast &e) {
        res = false;
    }
    
    return(res);

}

void
DSCStrategy::elsify_unwanted_keys(DSCUMap &dscdata) {

    // elsify_unwanted_keys
    DSCUMap::iterator it=dscdata.begin();
    
    boost::unordered_map<string,int> tmp_value;
    boost::unordered_map<string,int>::iterator it1;
    while ( it != dscdata.end() ) {
        // skip if map key (second part) is else
        // skip if map key (second part) is in the list of valid 
        // keys for this graph
        if ( ((*it).first.second == "else") || (keys_.find((*it).first.second) != keys_.end()) ) {
            it++;
            continue;
        }
        // Sum the values of the unknown keys 
        // and then delete the unknown key.
        // Don't forget we are in an iterator and so must post increment it
        tmp_value[(*it).first.first] += (*it).second;
        dscdata.erase(it++);
    }
    
    it1 = tmp_value.begin();
    for(; it1 != tmp_value.end(); it1++) {
        DSCKey tmp_key1((*it1).first,"else");
        if ((*it1).second !=  0) {
            dscdata[tmp_key1] += (*it1).second;
        }
    }

}

int
DSCStrategy::accum1d_to_count(const DSCUMap &dscdata) {

    int count=0;
    DSCUMap::const_iterator it=dscdata.begin();
    for ( ; it!=dscdata.end() ; it++){
        if ((*it).first.second == "-:SKIPPED_SUM:-") {
            continue;
        } else if ((*it).first.second == "-:SKIPPED:-") {
            count += (*it).second;
        } else {
            count++;
        }
    }
    return count;

}

void
DSCStrategy::merge_accum1d(const DSCUMap &dscdata, DSCUMap &accum){

    DSCKey tmp_key;
    tmp_key.first = "All";
    DSCUMap::const_iterator it=dscdata.begin();
    for ( ; it!=dscdata.end() ; it++){
        tmp_key.second = (*it).first.second;
        accum[tmp_key] += (*it).second;
    }

}

// convert a "2D" accum-type dataset to a "count-type"
// ie, count all input->{*}{k2} into output->{k2}
void
DSCStrategy::accum2d_to_count(const DSCUMap &dscdata, DSCUMap &count) {
    
    DSCUMap::const_iterator it=dscdata.begin();
    DSCKey tmp_key;
    while ( it != dscdata.end() ) {
        if ( (*it).first.second == "-:SKIPPED_SUM:-" ) { 
       
        } else if ( (*it).first.second == "-:SKIPPED:-" ) {
            tmp_key.first = "All";
            tmp_key.second = (*it).first.first;
            count[tmp_key] += (*it).second;
        } else {
            tmp_key.first = "All";
            tmp_key.second = (*it).first.first;
            count[tmp_key] += 1;
        }
        it++;
    }

}

// convert a "2D" accum-type dataset to a trace-type
// ie, sum all input->{*}{k2} into output->{k2}
void
DSCStrategy::accum2d_to_trace(const DSCUMap &dscdata, DSCUMap &trace){

    DSCUMap::const_iterator it=dscdata.begin();
    DSCKey tmp_key;
    while ( it != dscdata.end() ) {
        if ( (*it).first.second != "-:SKIPPED:-" ) {
            tmp_key.first = "All";
            tmp_key.second = (*it).first.first;
            trace[tmp_key] += (*it).second;
        }
        it++;
        
    }

}

void
DSCStrategy::swap_dimensions(const DSCUMap &dscdata, DSCUMap &swap) {

    DSCUMap::const_iterator it=dscdata.begin();
    DSCKey tmp_key;
    while ( it != dscdata.end() ) {
        tmp_key.first = (*it).first.second;
        tmp_key.second = (*it).first.first;
        swap[tmp_key] = (*it).second;
        it++;
    }
    if ( swap.size() != dscdata.size() ) cout << "Error swapping keys\n";

}

void
DSCStrategy::merge_accum2d(const DSCUMap &dscdata, DSCUMap &old) {

    DSCUMap::const_iterator it=dscdata.begin();
    for ( ; it != dscdata.end() ; it++ ) old[(*it).first] += (*it).second;

}

void
DSCStrategy::trim_accum2d(DSCUMap &dscdata) {

    // key order has been swapped before getting here
    // dscdata -> (TLD, QTYPE) -> count)
    multimap<int,string> r_map;
    multimap<int,string>::reverse_iterator ir;
    set<DSCKey> keys2delete;
    
    // iterate over allowed keys
    set<string>::iterator key = keys_.begin();
    for (; key != keys_.end() ; key++ ) {
        
        int n = 0;
        // iterate over dscdata and create multimap
        // r_map = count -> TLD (sorted))
        DSCUMap::iterator it=dscdata.begin();
        for (; it != dscdata.end() ; it++) {   
            if ((*it).first.second != (*key) ) continue;
            // use multimap to sort by value
            r_map.insert(pair<int,string>((*it).second,(*it).first.first));
        }
        // iterate over r_map
        ir=r_map.rbegin();
        for (; ir != r_map.rend() ; ir++) {
            // Create key_pair
            // TLD,allowed key
            DSCKey key_pair ((*ir).second,*key);
            // continue if n <= 1000 
            //  this means that we have seen 1000 or more (TLD,allowed key) pairs
            if ( ++n <= 1000 ) continue;
            // continue if key_pair is not found in dscdata
            if ( dscdata.find(key_pair) == dscdata.end() ) continue;
            // so now we have TLD,allowed key pairs above 1000
            // for each one add one to dscdata("-:SKIPPED:-", allowed key)  value
            DSCKey tmp_skip ("-:SKIPPED:-",*key);
            dscdata[tmp_skip]++;        
            // and add the value to a dscdata("-:SKIPPED_SUM:-", allowed key)  value
            DSCKey tmp_skip_sum ("-:SKIPPED_SUM:-",*key);
            dscdata[tmp_skip_sum] += dscdata[key_pair];
            // Add the (TLD,allowed key pair) to a set of keys to be deleted later
            keys2delete.insert(key_pair);
        }
        r_map.clear();
        // and finally delete the dscdata entries in set keys2delete
        set<DSCKey>::iterator it1 = keys2delete.begin();
        for (; it1 != keys2delete.end(); it1++) {
            dscdata.erase((*it1));
        }
        keys2delete.clear();
    }
    return;

}

void
DSCStrategy::open_file(string dtime[], ios_base::openmode open_mode) {

    bfs::path p_tmp;
    bfs::path p_tmp_file;
    bfs::path p_unique;
    bfs::path p_data_root = ".";
    bfs::path p_storage = dtime[0];
    bfs::path p_data_storage(p_data_root/p_storage);
    bfs::path p_filename(plot_name_ + ".dat");
    bfs::path p_dat_file(p_data_storage/p_filename);
    
    if ( open_mode & ios_base::app || open_mode & ios_base::trunc || open_mode & ios_base::out) {
        p_unique = bfs::unique_path();
        p_tmp_file = plot_name_ + ".dat." + p_unique.string();
        p_tmp = p_data_storage/p_tmp_file;
    }

    try {
        
        if (! bfs::exists(p_data_storage)) bfs::create_directory(p_data_storage);
        if (! bfs::exists(p_data_storage)) {
            cerr <<  "FS Error: " << p_data_storage << " directory could not be created." << endl;
            return;
        }
        if (! bfs::is_directory(p_data_storage)) {
            cerr <<  "FS Error: " <<  p_data_storage << " exists but is not a directory." << endl;
            return;
        }
        
        if ( open_mode & ios_base::app || open_mode & ios_base::trunc || open_mode & ios_base::out) {
            if (bfs::exists(p_dat_file)) bfs::copy_file(p_dat_file, p_tmp);
            datafile_.filename=p_dat_file.string();
            datafile_.tmpfilename=p_tmp.string();
            datafile_.open_mode = open_mode;
            datafile_.filestream.open(p_tmp, open_mode);
            if (! datafile_.filestream.is_open()) {
                cerr <<  "FS Error: " << p_tmp << " temporary file did not open." << endl;
                return;
            }
        } else if ( open_mode & ios_base::in ) {
            datafile_.filename=p_dat_file.string();
            datafile_.open_mode = open_mode;
            datafile_.filestream.open(p_dat_file, open_mode);
            if (bfs::exists(p_dat_file) && (! datafile_.filestream.is_open()))  {
                cerr <<  "FS Error: " << p_dat_file << " dat file did not open." << endl;
                return;
            }
        } else {
            cerr << "FS Error: un-supported file open mode" << endl;
        }
    }
    catch (const bfs::filesystem_error& ex) {
        cerr << "FS Error: " << ex.what() << endl;
    }
    return;

}

void
DSCStrategy::close_file() {

    try {
        datafile_.filestream.close();
        if (datafile_.open_mode & ios_base::app || datafile_.open_mode & ios_base::trunc || datafile_.open_mode & ios_base::out) {
            bfs::rename(datafile_.tmpfilename, datafile_.filename);
        }
    }
    catch (const bfs::filesystem_error& ex) {
        cerr << "FS Error: " << ex.what() << endl;
    }

}

int
DSCStrategy::get_plot_id(const string& plot, pqxx::work& pg_db_trans) {

    stringstream sql;
    pqxx::result r;
    
    try {
        sql.clear();
        sql.str("");
        sql << "SELECT id FROM dsc.dataset WHERE name = '" << plot << "'" << endl;
        r = pg_db_trans.exec(sql.str());
        if ( r.size() != 1 ) {
            cerr << "Error: Expected 1 plot with name " << plot << ", "
                 << "but found " << r.size() << endl;
            exit( EXIT_FAILURE );
        }
    }
    catch( runtime_error & e )
    {
        cerr << "Runtime error: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( std::exception & e )
    {
        cerr << "Exception: " << e.what() << sql.str() << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }
    return r[0][0].as<int>();

}

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   DSCStrategy.h
 */

// TODO(asap): Doxygen comments

#ifndef DSCSTRATEGY_H
#define	DSCSTRATEGY_H

#include <string>
#include <set>
#include <map>

#define BOOST_FILESYSTEM_NO_DEPRECATED
#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/unordered_map.hpp>

#include <dsc_types.h>

#include <pqxx/pqxx>

using namespace std;
namespace bfs = boost::filesystem;

class DSCStrategy {

public:
    
    DSCStrategy() {};
    DSCStrategy(string server, string name, string keys[], int keylength);
    
    virtual ~DSCStrategy() {};
    virtual void process_to_dat(const DSCUMap &, string dtime[]) = 0;
    virtual int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) = 0;
    virtual int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) = 0;
    virtual void process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &file_path) = 0;
    virtual void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans) = 0;
    virtual bool is_dat_file_multi_unit() = 0;
    
protected:
    
    set<string> keys_;
    string server_name_;
    string plot_name_;
    DSCdatfile datafile_;
    
    virtual void write_dat(const DSCUMap &, string dtime[]) = 0;
    
    void elsify_unwanted_keys(DSCUMap &dscdata);
    void accum2d_to_count(const DSCUMap &dscdata, DSCUMap &count);
    void accum2d_to_trace(const DSCUMap &dscdata, DSCUMap &trace);
    void swap_dimensions(const DSCUMap &dscdata, DSCUMap &swap);
    void merge_accum2d(const DSCUMap &dscdata, DSCUMap &old);
    void open_file(string dtime[], ios_base::openmode open_mode);
    void close_file();
    void trim_accum2d(DSCUMap  &dscdata);
    int accum1d_to_count(const DSCUMap &dscdata);
    void merge_accum1d(const DSCUMap &dscdata, DSCUMap &accum);
    bool is_valid_time(const string& time_string);    
    int get_plot_id(const string& plot, pqxx::work& pg_db_trans);
    void add_1_to_skipped(const DSCUMap &dscdata, DSCUMap &newdata);
    
};

#endif	/* DSCSTRATEGY_H */

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   DSCStrategyFactory.cpp
 */

// TODO(refactor): This should be re-factored and pick up the data from a config file. 
// TODO(refactor): A config file should also be used to specify which data to process and which to ignore.

#include <iostream>
#include <string>
#include <vector>
#include "dsc_types.h"
#include "DSCStrategyFactory.h"
#include "DSCStrategy.h"
#include "Format1DSCStrategy.h"
#include "Format2DSCStrategy.h"
#include "Format3aDSCStrategy.h"
#include "Format3bDSCStrategy.h"
#include "Format4DSCStrategy.h"

using namespace std;

DSCStrategyFactory::DSCStrategyFactory() {

}

DSCStrategyFactory::~DSCStrategyFactory() {

}

vector<DSCStrategy*>
DSCStrategyFactory::createStrategy(const string &server, const string &name, bool rssac) {

    vector<DSCStrategy*> return_v;

    //  Format 1
    if (name.compare("rcode") == 0) {
        string keys[1] = { "dummy" };
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name, keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("server_addr") == 0) {
        string keys[1] = { "dummy" };
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name, keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("qtype") == 0) {
        string keys[11] = { "1", "2", "5", "6", "12", "15", "28", "33",
                                "38", "255",  "else"};
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,11);
        return_v.push_back( strat );
        
        string keys1[8] = {"24", "25", "30", "43", "46", "47", "50", "48"};
        Format1DSCStrategy* strat1 = new Format1DSCStrategy(server,"dnssec_qtype",keys1,8);
        return_v.push_back( strat1 );
        return return_v;        
    
    } else if (name.compare("opcode") == 0) {
        string keys[6] = {"0", "1", "2", "4", "5", "else"};
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,6);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("edns_version") == 0) {
        string keys[3] = {"none", "0", "else"};
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,3);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("rd_bit") == 0) {
        string keys[2] = {"set", "clr"};
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,2);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("idn_qname") == 0) {
        string keys[2] = {"normal", "idn"};
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,2);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("do_bit") == 0) {
        string keys[2] = {"set", "clr"};
        Format1DSCStrategy* strat = new Format1DSCStrategy(server, name,keys,2);
        return_v.push_back( strat );
        return return_v;
    //  End Format 1

    //  Format 2
    } else if (name.compare("server_addr_vs_trans") == 0) {
        string keys[3] = {"dummy"};
        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,0);
        return_v.push_back( strat );
        return return_v; 
    
    } else if (name.compare("certain_qnames_vs_qtype") == 0) {
        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
                                "38", "255", "else"};
        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,11);
        return_v.push_back( strat );
        return return_v;
    
    } else if (name.compare("direction_vs_ipproto") == 0) {
        string keys[3] = {"icmp", "tcp", "udp"};
        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,3);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("chaos_types_and_names") == 0) {
        string keys[3] = {"hostname.bind", "version.bind", "other"};
        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,3);
        return_v.push_back( strat );
        return return_v;
    
    } else if (name.compare("pcap_stats") == 0) {
        string keys[1] = {"dummy"};
        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,0);
        return_v.push_back( strat );
        return return_v;

    } else if (name.compare("transport_vs_qtype") == 0) {
        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
                                "38", "255", "else"};
        Format2DSCStrategy* strat = new Format2DSCStrategy(server, name,keys,11);
        return_v.push_back( strat );
        return return_v;
    } else if ( name.compare("traffic_volume_queries") == 0 ||
        name.compare("traffic_volume_responses") == 0 ) {
        if (rssac) {
            string keys[1]= { "dummy" };
            Format2DSCStrategy* strat = new Format2DSCStrategy(server, name, keys,0);
            return_v.push_back( strat );
            return return_v;        
        } else {
            return return_v;
    }
    //  End Format 2 
       
    //  Format 3
    } else if (name.compare("client_subnet") == 0) {
        string keys[1] = {"dummy"};
        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server, "client_subnet_count",keys,0);
        return_v.push_back( strat );
        Format3bDSCStrategy* strat1 = new Format3bDSCStrategy(server, "client_subnet_accum",keys,0);
        return_v.push_back( strat1 );
        return return_v;
      
    } else if (name.compare("idn_vs_tld") == 0) {
        string keys[1] = {"dummy"};
        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server, name,keys,0);
        return_v.push_back( strat );
        return return_v;
    
    } else if (name.compare("ipv6_rsn_abusers") == 0) {
        string keys[1] = {"dummy"};
        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server, "ipv6_rsn_abusers_count",keys,0);
        return_v.push_back( strat );        
        Format3bDSCStrategy* strat1 = new Format3bDSCStrategy(server,"ipv6_rsn_abusers_accum",keys,0);
        return_v.push_back( strat1 );
        return return_v;

    } else if (name.compare("traffic_sizes_queries") == 0 ||
        name.compare("traffic_sizes_responses") == 0) {
        if (rssac) {
            string keys[1] = {"dummy"};
            Format3bDSCStrategy* strat = new Format3bDSCStrategy(server, name,keys,0);
            return_v.push_back( strat );
            return return_v;
        } else {
            return return_v;
        }

    } else if (name.compare("unique_sources") == 0) {
        if (rssac) {
            string keys[1] = {"dummy"};
            Format3bDSCStrategy* strat = new Format3bDSCStrategy(server, "unique_sources_raw",keys,0);
            return_v.push_back( strat );
            return return_v;    
        } else {
            return return_v;   
        }
    //  End Format 3
        
    //  Format 4
    } else if (name.compare("qtype_vs_tld") == 0) {
        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
                                "38", "255", "else"};
        Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,11);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("client_addr_vs_rcode") == 0) {
        string keys[1] = {"dummy"};
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"client_addr_vs_rcode_accum",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("qtype_vs_client_subnet") == 0) {
        string keys[1] = {"dummy"};
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"qtype_vs_client_subnet",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("qtype_vs_qnamelen") == 0) {
        string keys[1] = {"dummy"};
        Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("rcode_vs_replylen") == 0) {
        string keys[1] = {"dummy"};
        Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,0);
        return_v.push_back( strat );
        return return_v;       
        
    // } else if (name.compare("client_subnet_vs_tld") == 0) {
    //     string keys[1] = {"dummy"};
    //     Format4DSCStrategy* strat = new Format4DSCStrategy(server, name,keys,0);
    //     return_v.push_back( strat );
    //     return return_v;
        
    //  End format 4
        
    //  Multiple formats        
    } else if (name.compare("client_subnet2") == 0) {
        string keys[11] = {"ok", "non-auth-tld", "root-servers.net",
                                "localhost", "a-for-a", "a-for-root",
                                "rfc1918-ptr", "funny-qclass", "funny-qtype", 
                                "src-port-zero", "malformed"};
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"client_subnet2_trace",keys,11);
        return_v.push_back( strat );
        
        Format1DSCStrategy* strat1 = new Format1DSCStrategy(server,"client_subnet2_count",keys,11);
        return_v.push_back( strat1 );
        
        Format4DSCStrategy* strat2 = new Format4DSCStrategy(server,"client_subnet2_accum",keys,11);
        return_v.push_back( strat2 );
        return return_v;
        
    } else if (name.compare("dns_ip_version_vs_qtype") == 0) {
        string keys[11] = {"1", "2", "5", "6", "12", "15", "28", "33",
                                "38", "255", "else"};
        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"dns_ip_version",keys,11);
        return_v.push_back( strat );
        
        Format4DSCStrategy* strat1 = new Format4DSCStrategy(server,"dns_ip_version_vs_qtype",keys,11);
        return_v.push_back( strat1 );
        return return_v;
    //  End Multiple formats

    } else {
        // return empty vector
        return return_v;
    }

}

vector<DSCStrategy*>
DSCStrategyFactory::createStrategyDat(const string &server, const string &name) {

    vector<DSCStrategy*> return_v;
    string keys[1] = {"dummy"};

    //  Format 1    
    if (name.compare("rcode.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"rcode", keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("qtype.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"qtype",keys,0);
        return_v.push_back( strat );
        return return_v;        
    
    } else if (name.compare("dnssec_qtype.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"dnssec_qtype",keys,0);
        return_v.push_back( strat );
        return return_v;
    
    } else if (name.compare("opcode.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"opcode",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("edns_version.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"edns_version",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("rd_bit.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"rd_bit",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("idn_qname.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"idn_qname",keys,0);
        return_v.push_back( strat );
        return return_v;        
        
    } else if (name.compare("do_bit.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"do_bit",keys,0);
        return_v.push_back( strat );
        return return_v;
    //  End Format 1 

    //  Format 2
    } else if (name.compare("certain_qnames_vs_qtype.dat") == 0) {
        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"certain_qnames_vs_qtype",keys,0);
        return_v.push_back( strat );
        return return_v;
    
    } else if (name.compare("direction_vs_ipproto.dat") == 0) {
        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"direction_vs_ipproto",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("chaos_types_and_names.dat") == 0) {
        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"chaos_types_and_names",keys,0);
        return_v.push_back( strat );
        return return_v;
    
    } else if (name.compare("pcap_stats.dat") == 0) {
        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"pcap_stats",keys,0);
        return_v.push_back( strat );
        return return_v;

    } else if (name.compare("transport_vs_qtype.dat") == 0) {
        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"transport_vs_qtype",keys,0);
        return_v.push_back( strat );
        return return_v;
    //  End Format 2 

    //  Format 3
    } else if (name.compare("client_subnet_count.dat") == 0) {
        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server,"client_subnet_count",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("client_subnet_accum.dat") == 0) {
        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server,"client_subnet_accum",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("idn_vs_tld.dat") == 0) {
        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server,"idn_vs_tld",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("ipv6_rsn_abusers_count.dat") == 0) {
        Format3aDSCStrategy* strat = new Format3aDSCStrategy(server,"ipv6_rsn_abusers_count",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("ipv6_rsn_abusers_accum.dat") == 0) {
        Format3bDSCStrategy* strat = new Format3bDSCStrategy(server,"ipv6_rsn_abusers_accum",keys,0);
        return_v.push_back( strat );
        return return_v;
    //  End Format 3

    //  Format 4
    } else if (name.compare("qtype_vs_tld.dat") == 0) {
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"qtype_vs_tld",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("client_addr_vs_rcode_accum.dat") == 0) {
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"client_addr_vs_rcode_accum",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("qtype_vs_qnamelen.dat") == 0) {
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"qtype_vs_qnamelen",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("rcode_vs_replylen.dat") == 0) {
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"rcode_vs_replylen",keys,0);
        return_v.push_back( strat );
        return return_v;
    //  End format 4
        
    //  Multiple formats
    } else if (name.compare("client_subnet2_trace.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"client_subnet2_trace",keys,0);
        return_v.push_back( strat );
        return return_v;
    
    } else if (name.compare("client_subnet2_count.dat") == 0) {
        Format1DSCStrategy* strat = new Format1DSCStrategy(server,"client_subnet2_count",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("client_subnet2_accum.dat") == 0) {
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"client_subnet2_accum",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("dns_ip_version_vs_qtype.dat") == 0) {
        Format4DSCStrategy* strat = new Format4DSCStrategy(server,"dns_ip_version_vs_qtype",keys,0);
        return_v.push_back( strat );
        return return_v;
        
    } else if (name.compare("dns_ip_version.dat") == 0) {
        Format2DSCStrategy* strat = new Format2DSCStrategy(server,"dns_ip_version",keys,0);
        return_v.push_back( strat );
        return return_v;
    //  End Multiple formats

    } else {
        // return empty vector
        return return_v;
    }

}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   DSCStrategyFactory.h
 */

#ifndef DSCSTRATEGYFACTORY_H
#define	DSCSTRATEGYFACTORY_H

#include <string>
#include "dsc_types.h"
#include "DSCStrategy.h"

using namespace std;

class DSCStrategyFactory {
    DSCStrategyFactory();
    ~DSCStrategyFactory();
public:
    static vector<DSCStrategy*> createStrategy(const string &server, const string &name, bool rssac);
    static vector<DSCStrategy*> createStrategyDat(const string &server, const string &name);
};

#endif	/* DSCSTRATEGYFACTORY_H */

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#include <iostream>
#include <map>
#include <string>

#define BOOST_FILESYSTEM_NO_DEPRECATED
#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/tokenizer.hpp>
#include "dsc_types.h"
#include "Format1DSCStrategy.h"
#include "DSCStrategy.h"

#include "pqxx/pqxx"

using namespace std;
namespace bfs = boost::filesystem;

//
// time k1 v1 k2 v2 ...
//

Format1DSCStrategy::Format1DSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
    
}

void
Format1DSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string /*dtime*/[], int /*server*/, int /*node*/, pqxx::work* /*pg_db_trans*/) {
    
    if ( plot_name_ == "client_subnet2_count") {
        accum2d_to_count(orig_dscdata, new_dscdata);
    } else if ( plot_name_ == "client_subnet2_trace") {
        accum2d_to_trace(orig_dscdata, new_dscdata);
    } else if ( plot_name_ == "client_port_range" || 
                plot_name_ == "edns_bufsiz"  ||
                plot_name_ == "rcode" ||
                plot_name_ == "server_addr") {
        new_dscdata = orig_dscdata;
    } else {
        new_dscdata = orig_dscdata;
        elsify_unwanted_keys(new_dscdata);    
    }

}

void
Format1DSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {

    open_file(dtime, ios_base::app | ios_base::out);
    write_dat(dscdata, dtime);
    close_file();

}

int
Format1DSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

    return write_db(dscdata, dtime, server_id, node_id, pg_db_trans);

}

void
Format1DSCStrategy::write_dat(const DSCUMap &my_dscdata, string dtime[]){
    
    // Now store the new data in the file
    datafile_.filestream << dtime[1];
    
    DSCUMap::const_iterator it1=my_dscdata.begin();
    while ( it1 != my_dscdata.end() ) {
        if ( (*it1).second != 0 ) {
            datafile_.filestream << " " << (*it1).first.second << " " << (*it1).second;    
        }
        it1++;
    }

    datafile_.filestream << endl;

}

int
Format1DSCStrategy::process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) {

    DSCKey tmp_key;
    tmp_key.first = "All";
    typedef boost::tokenizer<boost::char_separator<char> > t_tokenizer;
    boost::char_separator<char> sep(" ");
    
    if ( dat_line.length() == 0 ) {
        cerr << endl << "Found empty line in dat file - skipped" << endl;
        return 1;
    }
    t_tokenizer tok(dat_line, sep);

    // Read the time and the alternating keys and values
    int i=0;
    t_tokenizer::iterator field=tok.begin();
    // Read time from first field
    time_strings[1] = *field;
    if (! is_valid_time(time_strings[1]) ) return 1;
    field++;
    for(; field!=tok.end();++field){
        i++;
        if ( i%2 != 0 ) tmp_key.second = *field;
        if ( i%2 == 0 ) {
            int tmp_value;
            stringstream(*field) >> tmp_value;
            counts[tmp_key] = tmp_value;
        }
    }
    return 0;

}

int
Format1DSCStrategy::write_db(const DSCUMap &my_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans){

    DSCUMap::const_iterator it1=my_dscdata.begin();
    stringstream sql;
    sql.clear();
    int plot_id = get_plot_id(plot_name_, pg_db_trans);
    sql.str("");    
    sql << "INSERT INTO dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\" (server_id, node_id, plot_id, starttime, key1, key2, value) values"; 
    while ( 1 ) {
        if ( (*it1).second != 0 ) {
            sql << " (" << server_id << ", " << node_id << ", " << plot_id 
                << ", timestamptz 'epoch' + " << dtime[1] 
                <<  " * interval '1 second', '" << (*it1).first.second << "', 'NONE', " 
                << (*it1).second << ")";

            it1++;
            if (it1 == my_dscdata.end() ) {
                sql << ";";
                break;
            } else {
                sql << ",";
            }
        } else {
            cerr << "Format 1 Error: Bad data at " << plot_name_ << " " << dtime[1] << ". Skipping..." << endl;
            return 1;
        }
    }
    try {
        pg_db_trans.exec(sql.str());
    }
    catch ( pqxx::unique_violation & e )
    {
        cerr << "Format 1 unique_violation error: " << e.what() << endl;
        return 1;
    }
    catch( runtime_error & e )
    {
        cerr << "Format 1 runtime error: " << e.what() << endl;
        return 1;
    }
    catch( std::exception & e )
    {
        cerr << "Format 1 exception: " << e.what() << sql.str() << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Format 1 Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }
    return 0;
}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   Format1DSCStrategy.h
 */

#ifndef FORMAT1DSCSTRATEGY_H
#define FORMAT1DSCSTRATEGY_H
#include <string>
#include <map>
#include "dsc_types.h"
#include"DSCStrategy.h"

using namespace std;

class Format1DSCStrategy: public DSCStrategy {

public:

    Format1DSCStrategy(string server, string name, string keys[], int keylength);
    virtual ~Format1DSCStrategy() {};

    void process_to_dat(const DSCUMap &, string dtime[]);
    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line);
    void process_dat_file(DSCUMap &/*counts*/, std::string /*time_strings*/[], const boost::filesystem::path &/*file_path*/) {};
    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
    bool is_dat_file_multi_unit() { return true; };
    
private:

    void write_dat(const DSCUMap &, string dtime[]);
    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);

};


#endif	/* FORMAT1DSCSTRATEGY_H */

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#define BOOST_FILESYSTEM_NO_DEPRECATED

#include <iostream>
#include <map>
#include <string>

#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/tokenizer.hpp>

#include "dsc_types.h"
#include "DSCStrategy.h"
#include "Format2DSCStrategy.h"

#include "pqxx/pqxx"

using namespace std;
namespace bfs = boost::filesystem;

Format2DSCStrategy::Format2DSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
    
}

void
Format2DSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string /*dtime*/[], int /*server*/, int /*node*/, pqxx::work* /*pg_db_trans*/) {

      if ( plot_name_.compare("traffic_volume_queries") == 0 || 
                plot_name_.compare("traffic_volume_responses") == 0  ||
                plot_name_.compare("traffic_sizes_queries") == 0 ||
                plot_name_.compare("traffic_sizes_responses") == 0 ) {
        new_dscdata = orig_dscdata;
    } else if ( ! keys_.empty() ) {
        new_dscdata = orig_dscdata;
        elsify_unwanted_keys(new_dscdata);    
    } else {
        swap_dimensions(orig_dscdata, new_dscdata);
    }

}

void
Format2DSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {
    
    open_file(dtime, ios_base::app | ios_base::out);
    write_dat(dscdata, dtime);
    close_file();

}

int
Format2DSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

     return write_db(dscdata, dtime, server_id, node_id, pg_db_trans);

}

void
Format2DSCStrategy::write_dat(const DSCUMap &my_dscdata, string dtime[]) {
    
    DSCUMap::const_iterator it;
    map<string,string> tmp_kv;
    map<string,string>::iterator does_key_exist;
    // Don't forget the map's key is a pair as well
    for ( it=my_dscdata.begin() ; it != my_dscdata.end(); it++ ){
        string t = static_cast<ostringstream*>( &(ostringstream() << (*it).second) )->str();
        does_key_exist = tmp_kv.find((*it).first.first); 
        if ( does_key_exist == tmp_kv.end() ) {
            tmp_kv[(*it).first.first] +=  (*it).first.second + ":" + t;
        } else {
            tmp_kv[(*it).first.first] +=  ":" + (*it).first.second + ":" + t;
        }
    }
    if (! tmp_kv.empty() ){
        datafile_.filestream << dtime[1];
        map<string,string>::iterator it1;
        for ( it1=tmp_kv.begin() ; it1 != tmp_kv.end(); it1++ ){
            datafile_.filestream << " " << (*it1).first << " " << (*it1).second;
        }
        datafile_.filestream << endl;
    }

}

int
Format2DSCStrategy::process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) {

    // reads a 2-level hash database WITH time dimension
    // ie: time k1 (k:v:k:v) k2 (k:v:k:v)
    // each line comes from a single XML array
    // Need to return time and DSCUMap for each line
    
    string k;
    DSCKey tmp_key;
    typedef boost::tokenizer<boost::char_separator<char> > t_tokenizer;
    boost::char_separator<char> sep(" ");
    boost::char_separator<char> sep1(":");

    if ( dat_line.length() == 0 )  {
        cerr << endl << "Found empty line in dat file - skipped" << endl;
        return 1;
    }

    t_tokenizer tok(dat_line, sep);
    t_tokenizer::iterator field=tok.begin();
    time_strings[1] = *field;
    if (! is_valid_time(time_strings[1]) ) return 1;
    field++;
    int i=0;
    for(; field!=tok.end();++field) {
        i++;
        if ( i%2 != 0 ) {
            tmp_key.first = *field;
            continue;
        }
        
        t_tokenizer tok1(*field, sep1);
        int j=0;
        for(t_tokenizer::iterator field1=tok1.begin(); field1 != tok1.end();++field1) {
            j++;
            if ( j%2 != 0 ) {
                tmp_key.second = *field1;
                continue;
            }

            int tmp_value;
            stringstream(*field1) >> tmp_value;
            counts[tmp_key] = tmp_value;
        }
    }
    return 0;

}

int
Format2DSCStrategy::write_db(const DSCUMap &my_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

    DSCUMap::const_iterator it1=my_dscdata.begin();
    stringstream sql;
    sql.clear();
    int plot_id = get_plot_id(plot_name_, pg_db_trans);
    sql.str("");

    sql << "INSERT INTO dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\" (server_id, node_id, plot_id, starttime, key1, key2, value) values";

    if (my_dscdata.empty()) {
      cerr << "Format 2 Error: Empty data set for " << plot_name_ << " " << dtime[1] << ". Skipping..." << endl;
      return 1;
    }

    while ( 1 ) {
        if ( (*it1).second != 0 ) {
            sql << " (" << server_id << ", " << node_id << ", " << plot_id
                << ", timestamptz 'epoch' + " << dtime[1] <<  " * interval '1 second', '"
                << (*it1).first.first << "','" << (*it1).first.second << "', " << (*it1).second << ")";

            it1++;
            if (it1 == my_dscdata.end() ) {
                sql << ";";
                break;
            } else {
                sql << ",";
            }
        } else {
            cerr << "Format 2 Error: Bad data at " << plot_name_ << " " << dtime[1] << ". Skipping..." << endl;
            return 1;
        }
    }

    try {
        pg_db_trans.exec(sql.str());
    }
    catch ( pqxx::unique_violation & e )
    {
        cerr << "Format 2 unique_violation error: " << e.what() << endl;
        return 1;
    } 
    catch( runtime_error & e )
    {
        cerr << "Format 2 runtime error: " << e.what() << endl;
        return 1;
    }
    catch( std::exception & e )
    {
        cerr << "Format 2 exception: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Format 2 Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }
    return 0;

}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#ifndef FORMAT2DSCSTRATEGY_H
#define FORMAT2DSCSTRATEGY_H
#include <string>
#include <map>
#include "dsc_types.h"
#include "DSCStrategy.h"
#include "pqxx/pqxx"

using namespace std;

class Format2DSCStrategy: public DSCStrategy {
    
public:

    Format2DSCStrategy(string server, string name, string keys[], int keylength);
    virtual ~Format2DSCStrategy() {};
    void process_to_dat(const DSCUMap &, string dtime[]);
    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line);
    void process_dat_file(DSCUMap &/*counts*/, std::string /*time_strings*/[], const boost::filesystem::path &/*file_path*/) {};
    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
    bool is_dat_file_multi_unit() { return true; };

private:

    void write_dat(const DSCUMap &, string dtime[]);
    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);

};

#endif	/* FORMAT2DSCSTRATEGY_H */

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#define BOOST_FILESYSTEM_NO_DEPRECATED

#include <iostream>
#include <map>
#include <string>

#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/algorithm/string.hpp>

#include "pqxx/pqxx"

#include "dsc_types.h"
#include "DSCStrategy.h"
#include "Format3aDSCStrategy.h"

//  Format3 has format k N(k)
//  it also contains data for all time not just 1 day

using namespace std;

Format3aDSCStrategy::Format3aDSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
    
}

void
Format3aDSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string /*dtime*/[], int /*server*/, int /*node*/, pqxx::work* /*pg_db_trans*/) {
	
    // Only called if the input channel was XML
    DSCKey tmp_key;
    tmp_key.first = "All";
    tmp_key.second = "All";
    new_dscdata[tmp_key] = accum1d_to_count(orig_dscdata);

}

void
Format3aDSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {
    
    open_file(dtime, ios_base::app | ios_base::out);
    write_dat(dscdata, dtime);
    close_file();

}

int
Format3aDSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

    return write_db(dscdata,dtime,server_id,node_id,pg_db_trans);

}

void
Format3aDSCStrategy::write_dat(const DSCUMap &dscdata, string dtime[]) {
    
    DSCUMap::const_iterator it = dscdata.begin();
    datafile_.filestream << dtime[1];
    datafile_.filestream << " " << (*it).second << endl;

}

int
Format3aDSCStrategy::process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line) {

    int tmp_value;
    DSCKey tmp_key;
    tmp_key.first = "All";
    vector<string> splitvec;
    vector<string>::iterator it;
    
    boost::split(splitvec, dat_line, boost::is_any_of(" "));
    if ( splitvec.size() != 2 ) {
        cerr << endl << "Found line containing: " << dat_line << " - skipped." << endl;
        return 1;
    }
    it = splitvec.begin();
    if(*it == "#MD5") return 1;
    tmp_key.second = (*it);
    time_strings[1] = (*it);
    if (! is_valid_time(time_strings[1]) ) return 1;
    it++;
    stringstream((*it)) >> tmp_value;
    
    counts[tmp_key] = tmp_value;
    splitvec.clear();
    return 0;

}

void
Format3aDSCStrategy::read_db(DSCUMap &old_dscdata, string dtime[],
                     int server_id, int node_id, pqxx::work& pg_db_trans) {

    DSCKey tmp_key;
    tmp_key.first = "All";
    int tmp_int = 0;
    string tmp_str;
    string key1 ("key1");
    string key2 ("key2");
    string value ("value");
    stringstream sql;
    int plot_id = get_plot_id(plot_name_, pg_db_trans);
    sql << "SELECT key1, value FROM dsc.format1_3 where starttime=to_timestamp('" << dtime[1] 
        << "') AND plot_id = '" << plot_id << "'" 
        << " AND node_id = '" << node_id << "'"
        << " AND server_id = '" << server_id << "';" << endl;

    try{

        const pqxx::result R = pg_db_trans.exec(sql.str());
        pqxx::result::const_iterator row;
        for (row = R.begin(); row != R.end(); ++row) {
            row[key1].to(tmp_str);
            row[value].to(tmp_int);
            tmp_key.second = tmp_str;
            old_dscdata[tmp_key] = tmp_int;
        }
    }
    catch( runtime_error & e )
    {
        cerr << "Format 3a runtime error: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( std::exception & e )
    {
        cerr << "Format 3a exception: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Format 3a Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }
}

int
Format3aDSCStrategy::write_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

    stringstream sql;
    int plot_id = get_plot_id(plot_name_, pg_db_trans);
    DSCUMap::const_iterator it = dscdata.begin();    
    sql << "INSERT INTO dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2]
        << "\" (server_id, node_id, plot_id, starttime, key1, key2, value) values ("
        << server_id << ", " << node_id << ", " << plot_id
        << ", timestamptz 'epoch' + " << dtime[1] <<  " * interval '1 second', '" 
        << dtime[1] << "', 'NONE', " << (*it).second << ");" << endl;
    try {
        (void)pg_db_trans.exec(sql.str());
    }
    
    catch ( pqxx::unique_violation & e )
    {
        cerr << "Format 3a unique_violation error: " << e.what() << endl;
        return 1;
    }   
    catch( runtime_error & e )
    {
        cerr << "Format 3a runtime error: " << e.what() << endl;
        return 1;
    }
    catch( std::exception & e )
    {
        cerr << "Format 3a exception: " << e.what() << endl;
        exit( EXIT_FAILURE );
    }
    catch( ... )
    {
        cerr << "Format 3a Unknown exception caught" << endl;
        exit( EXIT_FAILURE );
    }
    return 0;

}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   Format3DSCStrategy.h
 */

#ifndef FORMAT3aDSCSTRATEGY_H
#define FORMAT3aDSCSTRATEGY_H
#include <string>
#include <map>
#include "dsc_types.h"
#include "DSCStrategy.h"
#include "pqxx/pqxx"

using namespace std;

class Format3aDSCStrategy: public DSCStrategy {
    
public:

    Format3aDSCStrategy(string server, string name, string keys[], int keylength);
    virtual ~Format3aDSCStrategy() {};

    void process_to_dat(const DSCUMap &, string dtime[]);
    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    int process_dat_line(DSCUMap &counts, std::string time_strings[], std::string dat_line);
    void process_dat_file(DSCUMap &/*counts*/, std::string /*time_strings*/[], const boost::filesystem::path &/*file_path*/) {};
    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
    bool is_dat_file_multi_unit() { return true; };

private:

    void write_dat(const DSCUMap &dscdata, string dtime[]);    
    int write_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    void read_db(DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);

};

#endif	/* FORMAT3aDSCSTRATEGY_H */

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#define BOOST_FILESYSTEM_NO_DEPRECATED

#include <iostream>
#include <map>
#include <string>

#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/algorithm/string.hpp>

#include "pqxx/pqxx"

#include "dsc_types.h"
#include "DSCStrategy.h"
#include "Format3bDSCStrategy.h"

// Format3b has format k N(k)

using namespace std;

Format3bDSCStrategy::Format3bDSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {
    
}

void
Format3bDSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string dtime[], int /*server_id*/, int /*node_id*/, pqxx::work* pg_db_trans) {
    
    if (pg_db_trans == NULL) {
        open_file(dtime, ios_base::in);
        read_dat(new_dscdata);
        merge_accum1d(orig_dscdata,new_dscdata);
        close_file();
    }
    else {
        new_dscdata = orig_dscdata;
    }

}

void
Format3bDSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {

    // reopen dat file with trunc to write out merged data
    open_file(dtime, ios_base::trunc | ios_base::out);
    write_dat(dscdata,dtime);
    close_file();

}

int
Format3bDSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

    return write_db(dscdata,dtime,server_id,node_id,pg_db_trans);

}

void
Format3bDSCStrategy::write_dat(const DSCUMap &old_dscdata, string /*dtime*/[]) {

    DSCUMap::const_iterator it = old_dscdata.begin();
    for ( ; it != old_dscdata.end() ; it++) {
        datafile_.filestream << (*it).first.second << " " << (*it).second << endl;
    }

}

void
Format3bDSCStrategy::read_dat(DSCUMap &old_dscdata) {

    int tmp_value;
    DSCKey tmp_key;
    tmp_key.first = "All";
    string line;
    vector<string> splitvec;
    vector<string>::iterator it;
    while (getline(datafile_.filestream,line)) {
        boost::split(splitvec, line, boost::is_any_of(" "));
        if ( splitvec.size() != 2 ) {
            cerr << "Bad data skipped." << endl;
            continue;
        }
        it = splitvec.begin();
        tmp_key.second = (*it);
        if (tmp_key.second == "#MD5") continue;
        it++;
        stringstream((*it)) >> tmp_value;
        old_dscdata[tmp_key] = tmp_value;
        splitvec.clear();
    }

}

void
Format3bDSCStrategy::process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &/*file_path*/) {

    open_file(time_strings, ios_base::in);
    read_dat(counts);
    close_file();

}

int
Format3bDSCStrategy::write_db(const DSCUMap &old_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

    DSCUMap::const_iterator it1 = old_dscdata.begin();
    stringstream sql;
    stringstream tablename;
    sql.clear();
    int plot_id = get_plot_id(plot_name_, pg_db_trans);
    sql.str("");
    tablename << "dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\"";

    for ( ; it1 != old_dscdata.end() ; it1++ ) {    
        sql.clear();
        sql.str("");
        sql << "with upsert as ( update " << tablename.str() << " as d "
                << " set value= " << (*it1).second << " + e.value "
                << " from " << tablename.str() << " as e "
                << " where d.starttime=e.starttime and d.server_id=e.server_id and d.node_id=e.node_id and d.plot_id=e.plot_id and d.key1=e.key1 and d.key2=e.key2 "
                << " and e.server_id= " << server_id
                << " and e.node_id= " << node_id 
                << " and e.plot_id= " << plot_id 
                << " and e.starttime= " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')"
                << " and e.key1= '" << (*it1).first.first << "'"
                << " and e.key2= '" << (*it1).first.second  << "'"
                << " returning d.* )"
                << " insert into " << tablename.str()
                << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
                << " select " << server_id << " , " << node_id << " , " << plot_id << " , " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')" << " , '" << (*it1).first.first << "', '"  << (*it1).first.second << "', " << (*it1).second
                << " where not exists (select 1 from upsert);";
        try {       
            pg_db_trans.exec(sql.str());
        }
        catch ( pqxx::unique_violation & e )
        {
            cerr << "Format 3b unique_violation error: " << e.what() << endl;
            return 1;
        }   
        catch( runtime_error & e )
        {
            cerr << "Format 3b write_db runtime error: " << e.what() << endl;
            return 1;
        }
        catch( std::exception & e )
        {
            cerr << "Format 3b write_db exception: " << e.what() << endl;
            exit( EXIT_FAILURE );
        }
        catch( ... )
        {
            cerr << "Format 3b write_db Unknown exception caught" << endl;
            exit( EXIT_FAILURE );
        }
    }
    return 0;

}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   Format3DSCStrategy.h
 */

#ifndef FORMAT3bDSCSTRATEGY_H
#define FORMAT3bDSCSTRATEGY_H
#include <string>
#include <map>
#include "dsc_types.h"
#include "DSCStrategy.h"
#include "pqxx/pqxx"

using namespace std;

class Format3bDSCStrategy: public DSCStrategy {
    
public:

    Format3bDSCStrategy(string server, string name, string keys[], int keylength);
    virtual ~Format3bDSCStrategy() {};

    void process_to_dat(const DSCUMap &, string dtime[]);
    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    int process_dat_line(DSCUMap &/*counts*/, std::string /*time_strings*/[], std::string /*dat_line*/) { return 0; };
    void process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &file_path);
    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
    bool is_dat_file_multi_unit() { return false; };

private:

    void write_dat(const DSCUMap &, string dtime[]);
    void write_count_dat(int count, string dtime[]);    
    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    void write_count_db(int count, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    void read_dat(DSCUMap &);

};

#endif	/* FORMAT3bDSCSTRATEGY_H */

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

#include <iostream>
#include <map>
#include <string>
#include <vector>

#define BOOST_FILESYSTEM_NO_DEPRECATED
#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/algorithm/string.hpp>
#include <boost/foreach.hpp>

#include "dsc_types.h"
#include "DSCStrategy.h"
#include "Format4DSCStrategy.h"

#include "pqxx/pqxx"

// Format4 has format j k N(j,k)

using namespace std;
namespace bfs = boost::filesystem;

Format4DSCStrategy::Format4DSCStrategy(string server, string name, string keys[], int keylength) : DSCStrategy(server, name, keys, keylength) {

}

void
Format4DSCStrategy::preprocess_data(const DSCUMap &orig_dscdata, DSCUMap &new_dscdata, string dtime[], int /*server_id*/, int /*node_id*/, pqxx::work* pg_db_trans) {

    DSCUMap tmp_dscdata;

    // Call the munger
    // It depends on m_name - the if may be better in the factory.
    if ( plot_name_ == "client_subnet2_accum" || plot_name_ == "qtype_vs_tld" || plot_name_ == "client_addr_vs_rcode_accum") {
        swap_dimensions(orig_dscdata, tmp_dscdata);
    } else if ( plot_name_ == "dns_ip_version_vs_qtype" ) {
        tmp_dscdata = orig_dscdata;
        elsify_unwanted_keys(tmp_dscdata);  
    } else {
        tmp_dscdata = orig_dscdata;
    }
    
    // If we are writing to a dat file then we need to merge data
    // If we are writing to the database then we will use a CTE upsert
    if (pg_db_trans == NULL) {
        open_file(dtime, ios_base::in);   
        // Read the old dat file in as we need to merge old and new data
        read_dat(new_dscdata);   
        close_file();
        merge_accum2d(tmp_dscdata, new_dscdata);
        // Trimmer: runs once per hour of xml data
        if ( plot_name_ == "client_subnet2_accum" || plot_name_ == "qtype_vs_tld" || plot_name_ == "client_subnet_vs_tld") {
                int tmp_value;
                stringstream(dtime[1]) >> tmp_value;
                if ((59*60) == (tmp_value % 3600)) {
                        trim_accum2d(new_dscdata);
                }
        } 
    }  else {
        new_dscdata = tmp_dscdata;
    }
    
}

void
Format4DSCStrategy::process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &/*file_path*/) {

    open_file(time_strings, ios_base::in);
    read_dat(counts);
    close_file();

}

void
Format4DSCStrategy::process_to_dat(const DSCUMap &dscdata, string dtime[]) {

    open_file(dtime, ios_base::trunc | ios_base::out);
    write_dat(dscdata,dtime);
    close_file();

}

int
Format4DSCStrategy::process_to_db(const DSCUMap &dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans) {

    return write_db(dscdata,dtime, server_id, node_id, pg_db_trans);

}

void
Format4DSCStrategy::write_dat(const DSCUMap &old_dscdata, string /*dtime*/[]){

    DSCUMap::const_iterator it1=old_dscdata.begin();
    for ( ; it1 != old_dscdata.end() ; it1++ ) {
        datafile_.filestream << (*it1).first.first << " " << (*it1).first.second << " " << (*it1).second << endl;
    }

}

void
Format4DSCStrategy::read_dat(DSCUMap &old_dscdata) {

    string token;
    int tmp_value;
    DSCKey tmp_key;
    string line;
    vector<string> splitvec;
    vector<string>::iterator it;
    while (getline(datafile_.filestream,line)) {
        
        stringstream lineStream(line);
        
        lineStream >> tmp_key.first;
        if (tmp_key.first == "#MD5") continue;
        lineStream >> tmp_key.second;
        lineStream >> tmp_value;
        if (! lineStream.eof()) {
            cerr << "Bad data reading" << plot_name_ << "dat file" << endl;
            continue;
        }
        old_dscdata[tmp_key] = tmp_value;
    }

}

int
Format4DSCStrategy::write_db(const DSCUMap &old_dscdata, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans){

    stringstream sql;
    stringstream tablename;
    DSCUMap::const_iterator it1=old_dscdata.begin();
    sql.clear();
    int plot_id = get_plot_id(plot_name_, pg_db_trans);
    sql.str("");    
    tablename << "dsc.\"data_" << server_name_ << "_" << plot_name_ << "_" << dtime[2] << "\"";
    for ( ; it1 != old_dscdata.end() ; it1++ ) {    
        sql.clear();
        sql.str("");
        sql << "with upsert as ( update " << tablename.str() << " as d "
                << " set value= " << (*it1).second << " + e.value "
                << " from " << tablename.str() << " as e "
                << " where d.starttime=e.starttime and d.server_id=e.server_id and d.node_id=e.node_id and d.plot_id=e.plot_id and d.key1=e.key1 and d.key2=e.key2 "
                << " and e.server_id= " << server_id
                << " and e.node_id= " << node_id 
                << " and e.plot_id= " << plot_id 
                << " and e.starttime= " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')"
                << " and e.key1= '" << (*it1).first.first << "'"
                << " and e.key2= '" << (*it1).first.second  << "'"
                << " returning d.* )"
                << " insert into " << tablename.str()
                << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
                << " select " << server_id << " , " << node_id << " , " << plot_id << " , " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD')" << " , '" << (*it1).first.first << "', '"  << (*it1).first.second << "', " << (*it1).second
                << " where not exists (select 1 from upsert);";    
        try {
            pg_db_trans.exec(sql.str());
        }
        catch ( pqxx::unique_violation & e ) {
            cerr << "Format 4 unique_violation error: " << e.what() << endl;
            return 1;
        }
        catch( runtime_error & e )
        {
            cerr << "Format 4 write_db Runtime error: " << e.what() << endl;
            return 1;
        }
        catch( std::exception & e )
        {
            cerr << "Format 4 write_db exception: " << e.what() << endl;
            exit( EXIT_FAILURE );
        }
        catch( ... )
        {
            cerr << "Format 4 write_db Unknown exception caught" << endl;
            exit( EXIT_FAILURE );
        }
    }
    
    // Trimmer: runs once per hour of xml data
    int debug = 0;
    int tmp_value;
    stringstream(dtime[1]) >> tmp_value;
    if ((59*60) == (tmp_value % 3600)) {
        if ( debug == 1 ) cout << "Trimmer Started." << endl;
        if ( plot_name_ == "client_subnet2_accum" || plot_name_ == "qtype_vs_tld" || plot_name_ == "client_subnet_vs_tld") {
            if ( debug == 1 ) cout << "Trimmer found plot " << plot_name_ << endl;
            
            
            // first create a string to hold all the common query parameters
            stringstream sql_query_params;
            sql_query_params << " and server_id= " << server_id
                             << " and node_id= " << node_id 
                             << " and plot_id= " << plot_id 
                             << " and starttime= " << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD') ";
            stringstream sql_ignore_skipped;
            sql_ignore_skipped << " key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') ";
                  
            // Now figure out all the key2s which need trimming and loop over them
            string key2totrim;
            string key2 ("key2");
            sql.clear();
            sql.str("");
            sql << "select key2 from (select key2, count(*) from " << tablename.str()
                << " where " << sql_ignore_skipped.str() << sql_query_params.str()
                << " group by key2) as to_trim where count > 1000;";
            if ( debug == 1 ) cout << "Trimmer executed: " << sql.str() << endl << endl;
            try {
                 const pqxx::result R = pg_db_trans.exec(sql.str());
                 if ( debug == 1 ) cout << "Trimmer got back " << R.size() << " rows." << endl << endl;
                 pqxx::result::const_iterator row;
                 for (row = R.begin(); row != R.end(); ++row) {
                     row[key2].to(key2totrim);
                     stringstream sql1, sql2, sql3, sql4, sql5, sql6;
                     string skipped_str ("skipped");
                     string skipped_sum_str ("skipped_sum");                 
                     int skipped = 0;
                     int skipped_sum = 0;
                     int skipped_new = 0;
                     int skipped_sum_new = 0;
                                                          
                     // Calculate the new values - i.e. row count above 1000    
                     sql1 << "select sum(value) as skipped_sum, count(value) as skipped"
                          << " from "
                                << " (select key1, value from " << tablename.str()
                                << " where key2= '" << key2totrim << "' "
                                << " and " << sql_ignore_skipped.str() << sql_query_params.str() 
                                << " order by value desc offset 1000) as rowstotrim;";
                     if ( debug == 1 ) cout << "Trimmer executed: " << sql1.str() << endl << endl;
                     const pqxx::result R1 = pg_db_trans.exec(sql1.str());
                     if ( debug == 1 ) cout << "Trimmer got back " << R1.size() << " rows." << endl << endl;
                     pqxx::result::const_iterator row1;
                     for (row1 = R1.begin(); row1 != R1.end(); ++row1) {
                         row1[skipped_str].to(skipped_new);
                         row1[skipped_sum_str].to(skipped_sum_new);
                     }
                     if ( debug == 1 ) cout << "Trimmer skipped_new = " << skipped_new << ", skipped_sum_new = " << skipped_sum_new << endl << endl;
                     
                     // Pull the existing values out of the database for skipped and skipped sum
                     // and update or insert the new ones
                     sql2 << "select value as skipped_sum from " << tablename.str() 
                          << " where key2= '" << key2totrim 
                          << "' and key1='-:SKIPPED_SUM:-'"
                          << sql_query_params.str() << ";";
                     if ( debug == 1 ) cout << "Trimmer executed: " << sql2.str() << endl << endl;
                     const pqxx::result R2 = pg_db_trans.exec(sql2.str());
                     if ( debug == 1 ) cout << "Trimmer got back " << R2.size() << " rows." << endl << endl;
                     if (R2.size() > 1 ) {
                        cerr << "Format 4 write_db Expected 0 or 1 skipped_sum records. Found " << R2.size() << endl;
                        exit( EXIT_FAILURE );                     
                     }                    
                     if (R2.size() == 1) {
                         R2.begin()[skipped_sum_str].to(skipped_sum);
                         if ( debug == 1 ) cout << "Trimmer skipped_sum existing= " << skipped_sum << endl << endl;
                         skipped_sum = skipped_sum + skipped_sum_new;
                         if ( debug == 1 ) cout << "Trimmer skipped_sum updated= " << skipped_sum << endl << endl;
                         // Now update them in the database
                         sql4 << "update " << tablename.str()
                              << " set value = " << skipped_sum 
                              << " where key2='" << key2totrim << "' "
                              << " and key1='-:SKIPPED_SUM:-'"
                              << sql_query_params.str() << ";";
                         if ( debug == 1 ) cout << "Trimmer executed: " << sql4.str() << endl << endl;
                     }
                     else {
                        sql4 << " insert into " << tablename.str()
                             << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
                             << " values (" << server_id << " , " 
                             << node_id << " , " 
                             << plot_id << " , " 
                             << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD') " 
                             << ", '-:SKIPPED_SUM:-', '"  
                             << key2totrim << "', " 
                             << skipped_sum_new << ");";  
                        if ( debug == 1 ) cout << "Trimmer executed: " << sql4.str() << endl << endl;
                     }
                     const pqxx::result R10 = pg_db_trans.exec(sql4.str());
                     if ( debug == 1 ) cout << "Trimmer affected " << R10.affected_rows() << " rows." << endl << endl;
                     sql3 << "select value as skipped from " << tablename.str() 
                          << " where key2= '" << key2totrim 
                          << "' and key1='-:SKIPPED:-'"
                          << sql_query_params.str() << ";";
                     if ( debug == 1 ) cout << "Trimmer executed: " << sql3.str() << endl << endl;
                     const pqxx::result R3 = pg_db_trans.exec(sql3.str());
                     if ( debug == 1 ) cout << "Trimmer got back " << R3.size() << " rows." << endl << endl;
                     pqxx::result::const_iterator row3;
                     if (R3.size() > 1 ) {
                        cerr << "Format 4 write_db Expected 0 or 1 skipped records. Found " << R3.size() << endl;
                        exit( EXIT_FAILURE );                     
                     }                    
                     if (R3.size() == 1) {
                         R3.begin()[skipped_str].to(skipped);
                         if ( debug == 1 ) cout << "Trimmer skipped existing = " << skipped << endl << endl;
                         skipped = skipped + skipped_new;
                         if ( debug == 1 ) cout << "Trimmer skipped updated= " << skipped << endl << endl;
                         sql5 << "update " << tablename.str()
                              << " set value = " << skipped 
                              << " where key2='" << key2totrim << "' "
                              << " and key1='-:SKIPPED:-'"
                              << sql_query_params.str() << ";";
                         if ( debug == 1 ) cout << "Trimmer executed: " << sql5.str() << endl << endl;
                     }
                     else {
                         sql5 << " insert into " << tablename.str()
                             << " (server_id, node_id, plot_id, starttime, key1, key2, value) "
                             << " values (" << server_id << " , " 
                             << node_id << " , " 
                             << plot_id << " , " 
                             << "to_timestamp('" << dtime[0]  << "', 'YYYYMMDD') " 
                             << ", '-:SKIPPED:-', '"  
                             << key2totrim << "', " 
                             << skipped_new << ");";             
                         if ( debug == 1 ) cout << "Trimmer executed: " << sql5.str() << endl << endl;
                     }                           
                     const pqxx::result R11 = pg_db_trans.exec(sql5.str());
                     if ( debug == 1 ) cout << "Trimmer affected " << R11.affected_rows() << " rows." << endl << endl;

                     // Now we need to delete all the skipped keys
                     sql6 << "delete from " << tablename.str() 
                          << "where key2='" << key2totrim << "' "
                          << sql_query_params.str()
                          << " and key1 in "
                                << " (select key1 from " << tablename.str()
                                << " where key2= '" << key2totrim << "' "
                                << " and  " << sql_ignore_skipped.str() << sql_query_params.str() 
                                << " order by value desc offset 1000);";
                     if ( debug == 1 ) cout << "Trimmer executed: " << sql6.str() << endl << endl;
                     
                     const pqxx::result R12 = pg_db_trans.exec(sql6.str());
                     if ( debug == 1 ) cout << "Trimmer affected " << R12.affected_rows() << " rows." << endl << endl;
                }                  
            }
            catch ( pqxx::unique_violation & e ) {
                cerr << "Format 4 unique_violation error: " << e.what() << endl;
                return 1;
            }
            catch( runtime_error & e )
            {
                cerr << "Format 4 write_db Runtime error: " << e.what() << endl;
                return 1;
            }
            catch( std::exception & e )
            {
                cerr << "Format 4 write_db exception: " << e.what() << endl;
                exit( EXIT_FAILURE );
            }
            catch( ... )
            {
                cerr << "Format 4 write_db Unknown exception caught" << endl;
                exit( EXIT_FAILURE );
            }
        } 
    }
    return 0;

}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   Format4DSCStrategy.h
 */

#ifndef FORMAT4DSCSTRATEGY_H
#define FORMAT4DSCSTRATEGY_H

#include <string>
#include <map>
#include "dsc_types.h"
#include "DSCStrategy.h"
#include "pqxx/pqxx"

using namespace std;

class Format4DSCStrategy: public DSCStrategy {

public:

    Format4DSCStrategy(string server, string name, string keys[], int keylength);
    virtual ~Format4DSCStrategy() {};

    void process_to_dat(const DSCUMap &, string dtime[]);
    int process_to_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    int process_dat_line(DSCUMap &/*counts*/, std::string /*time_strings*/[], std::string /*dat_line*/) { return 0; };
    void process_dat_file(DSCUMap &counts, std::string time_strings[], const boost::filesystem::path &file_path);
    void preprocess_data(const DSCUMap &, DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work* pg_db_trans);
    bool is_dat_file_multi_unit() { return false; };

private:
	
    void write_dat(const DSCUMap &, string dtime[]);
    int write_db(const DSCUMap &, string dtime[], int server_id, int node_id, pqxx::work& pg_db_trans);
    void read_dat(DSCUMap &);

};

#endif	/* FORMAT4DSCSTRATEGY_H */

if DM
D = dsc-extractor
P = rssac_server.py rssac_server/rssac.py rssac_server/threadpool.py
S = rssacd
else
D =
P =
S =
endif

hhconfdir = @sysconfdir@/hedgehog

bin_PROGRAMS = $(D)

dsc_extractor_SOURCES = dsc-extractor.cpp \
                          DSCDataManager.cpp \
                          DSCIOManager.cpp \
                          DSCStrategyFactory.cpp \
                          DSCStrategy.cpp \
                          Format1DSCStrategy.cpp \
                          Format2DSCStrategy.cpp \
                          Format3aDSCStrategy.cpp \
                          Format3bDSCStrategy.cpp \
                          Format4DSCStrategy.cpp \
                          DSCDataManager.h \
                          DSCStrategyFactory.h \
                          DSCStrategy.h \
                          DSCIOManager.h \
                          Format1DSCStrategy.h \
                          Format2DSCStrategy.h \
                          Format3aDSCStrategy.h \
                          Format3bDSCStrategy.h \
                          Format4DSCStrategy.h \
                          dsc_types.h

dsc_extractor_CXXFLAGS = @LIBPQXX_INCLUDES@ @BOOST_CPPFLAGS@ -I$(top_srcdir)/src/libdsc

dsc_extractor_LDADD = @BOOST_SYSTEM_LDFLAGS@ \
                          @BOOST_SYSTEM_LIBS@ \
                          @BOOST_FILESYSTEM_LDFLAGS@ \
                          @BOOST_FILESYSTEM_LIBS@ \
                          @BOOST_PROGRAM_OPTIONS_LDFLAGS@ \
                          @BOOST_PROGRAM_OPTIONS_LIBS@ \
                          @LIBPQXX_LIBS@

python_PYTHON = $(P)
sbin_SCRIPTS = $(S)
EXTRA_DIST = rssac_server.in rssacd.in

do_subst_hh = sed -e 's|@HEDGEHOGYAML@|$(hhconfdir)|' -e "s|@DB_VERSION@|@DB_SCHEMA_VERSION@|"
do_subst_python = sed -e 's|@PYTHON_EXE@|$(PYTHON)|' -e 's|@HHPYSITEDIR@|$(pythondir)|'

rssac_server.py: $(srcdir)/rssac_server/rssac_server.in Makefile
	$(do_subst_hh) < $(srcdir)/rssac_server/rssac_server.in > rssac_server.py
	
rssacd: $(srcdir)/rssac_server/rssacd.in Makefile
	$(do_subst_python) < $(srcdir)/rssac_server/rssacd.in > rssacd

dist_man_MANS = rssacd.8 \
                dsc-extractor.1

rssacd.8: rssacd rssac_server.py rssac_server/rssac.py rssac_server/threadpool.py Makefile
	chmod 755 rssacd
	PYTHONPATH=${srcdir}/rssac_server $(HELP2MAN) --version-string=@PACKAGE_VERSION@ -N -s 8 -n "This listens for DNS Notify messages and records the zone size and the time for propogation of zones. This is only needed for the generation of RSSAC reports" ./rssacd -o rssacd.8

dsc-extractor.1: dsc-extractor Makefile
	$(HELP2MAN) -h -h --version-string=@PACKAGE_VERSION@ -N -s 1 --no-discard-stderr -n "Application to extract DSC data from XML or DAT files and insert it in to a Postgresql database. Users are recommended to use the refile_and_grok wrapper script in preference to using this exectutable directly." ./dsc-extractor -o dsc-extractor.1

dist_sysconf_DATA = rssacd.upstart \
                    rssac_server/rssacd.conf

rssacd.upstart: $(srcdir)/rssac_server/rssacd.upstart.in Makefile
	sed -e 's|@HEDGEHOGYAML@|$(hhconfdir)|' -e 's|@SBIN@|$(sbindir)|' < $(srcdir)/rssac_server/rssacd.upstart.in > rssacd.upstart

install-data-hook:
	mkdir -p ${DESTDIR}${sysconfdir}/init
	mv ${DESTDIR}${sysconfdir}/rssacd.upstart ${DESTDIR}${sysconfdir}/init/rssacd.conf
	mv ${DESTDIR}${sysconfdir}/rssacd.conf ${DESTDIR}${hhconfdir}/rssacd.conf
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/// Application which can insert data from DSC XML and DAT files into
/// a PostgreSQL database. It can also convert from XML to DAT.

#include "config.h"
#include <iostream>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

#include "dsc_types.h"
#include "DSCIOManager.h"

#define LOCK_FILE       "dsc-extractor.lock"

using namespace std;

/// \brief Usage function that prints usage information.
static void
usage (void)
{
    fprintf(stderr, "Usage: dsc-extractor [OPTION]...\n");
    fprintf(stderr,
        "Supported options:\n"
                "  -h                   Show this usage. \n"
                "  -i                   Input source: [XML|DAT]. Default XML. \n"
                "  -o                   Output destination: [DAT|PG_DB]. Default DAT. \n"
                "  -c                   Connection string for DB. Required. \n"
                "  -s                   Start date for data file processing. Default is to process all data. \n"
                "  -r                   Disable processing of rssac data. Default is to process all data. \n");
    fprintf(stderr, "Version %s. Report bugs to <%s>.\n",
            PACKAGE_VERSION, PACKAGE_BUGREPORT);  
}

/// \brief Main function parses command line and calls the DSCIOManager to load 
/// and convert the data.
int
main(int argc, char *argv[]) {
    int pfp;
    char str[32];
    pid_t pid;
    pid = getpid();
    pfp=open(LOCK_FILE, O_RDWR|O_CREAT, 0640);
    if ( pfp < 0 ) {
        cerr << "Error: Unable to open pid file " << LOCK_FILE << endl;
        exit(1);
    }
    if ( lockf(pfp,F_TLOCK,0) < 0 ) {
        cerr << "Error: Unable to lock pid file " << LOCK_FILE << endl;
        close(pfp);
        exit(0);
    }
    if ( snprintf(str, sizeof(str), "%lu\n", (unsigned long) pid) < 0 ) {
        cerr << "Error: Unable to convert pid to string.\n" << endl;
        close(pfp);
        exit(1);
    }
    if ( write(pfp,str,strlen(str)) < 0 ) {
        cerr << "Error: Unable to write to pid file " << LOCK_FILE << endl;
        close(pfp);
        exit(1);
    }
    
    // default settings that make this code work as a drop in replacement for
    // the original dsc-xml-extractor.pl 
    enum INPUT_OUTPUT input  = XML;
    enum INPUT_OUTPUT output = DAT;
    // connection string to be used if the output is to a database
    string conn_string;
    // string used to determine which date to start processing files from
    string processing_start_date_string;
    // process rssac data by default if present
   bool rssac=true; 
    
    // parse command line options
    int c;
    while ((c = getopt(argc, argv, "hi:o:c:s:r")) != -1) {
        switch (c) {
        case 'h':
            usage();
            exit(0);
        case 'i': {
            string input_string = optarg;
            if (input_string.compare("XML") == 0) {
                input=XML;
            } else if (input_string.compare("DAT") == 0) {
                input=DAT;
            } else {
                cout << "Error: Invalid -i value" << endl;
                usage();
                exit(1);
            }
            break;
        }
        case 'o': {        
            string output_string = optarg;
            if (output_string.compare("DAT") == 0) {
                output=DAT;
            } else if (output_string.compare("PG_DB") == 0) {
                output=PG_DB;
            } else {
                cout << "Error: Invalid -o value" << endl;
                usage();
                exit(1);
            }
            break;
        }    
        case 'c': {
            conn_string = optarg;
            if (conn_string.empty()) {
                cout << "Error: A connection string must be supplied with the -c flag." << endl;
                usage();
                exit(1);
            }
            break;
        }
        case 's': {
            processing_start_date_string = optarg;
            if (processing_start_date_string.empty()) {
                cout << "A processing start date in format YYYY-MM-DD must be provided when using -s." << endl;
                exit(1);
            }
            break;
        }
       case 'r': {
            rssac=false;
            cout << "RSSAC data in the XML will NOT be processed" << endl;
            break;
        }
        case '?': {
            usage();
            exit(1);
        }        
        default: {
            // nothing to do here just go with defaults
            break;
        }    
        }
    }
    
    if (input == output) {
        cout << "Error: Input and output cannot be the same. \n";
        exit(1);
    }   
    if ( input != XML && ! processing_start_date_string.empty()) {
        cerr << "Error: Processing start date option is only supported for input type of XML. Exiting." << endl;
        exit(1);
    }    
    
    try {
        DSCIOManager* data_io_manager = new DSCIOManager(input, output, conn_string, processing_start_date_string, rssac);
        data_io_manager->dsc_activate_ouput_destination();
        data_io_manager->dsc_import_input_from_source();
        data_io_manager->dsc_deactivate_ouput_destination();
        delete data_io_manager;
    }
    catch (std::exception &e) {
        cerr << "Error in libdsc: " << e.what() << endl;
        return 1;
    }
    
    if ( unlink(LOCK_FILE) < 0 ) {
        cerr << "Error: Unable to unlink pid file " << LOCK_FILE << endl;
    }
    return 0;
}
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* 
 * File:   dsc_types.h
 */

#ifndef DSC_TYPES_H
#define	DSC_TYPES_H
#define BOOST_FILESYSTEM_NO_DEPRECATED

#include <string>

#include <boost/filesystem.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/unordered_map.hpp>

#include <pqxx/pqxx>

using namespace std;
using namespace boost;
namespace bfs = boost::filesystem;

/// \brief Numeration to define valid input sources and output destinations.
enum INPUT_OUTPUT {
    XML,
    DAT,
    PG_DB
};

/// \brief Typedef defining the key pair that is used in the data map.
typedef pair<string, string> DSCKey;

/// \brief Ensure that there is a hash function for use in
/// boost::unordered_map
namespace boost {
    template <>
            class hash<DSCKey> {
            public:
            size_t operator() ( const DSCKey &akey ) const
            {
                size_t v=0;
                boost::hash_combine(v, akey.first);
                boost::hash_combine(v, akey.second);
                return v;
            }
    };
}

/// \brief Typedef defining a map that can hold any type of DSC data.
typedef boost::unordered_map<DSCKey,int> DSCUMap;

/// \brief Structure to hold information about dat files that are being manipulated
/// by the DSC code.
struct dsc_dat_file {
    string filename;
    string tmpfilename;
    bfs::fstream filestream;
    ios_base::openmode open_mode;
};
typedef dsc_dat_file DSCdatfile;


/// \brief Max number of strategies for a given dsc xml array name.
/// 4 is fine for all the current ones.
//#define MAXSTRATEGIES 4

#endif	/* DSC_TYPES_H */
#!/usr/bin/env python
# 
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

#
# Original developed by john.bond@icann.org
# Modified for use with Hedgehog by Sinodun IT (www.sinodun.com)

import argparse
import os
import glob
import yaml
import time
import sys
import dns.query
import dns.message
import dns.rdatatype
import dns.exception
import logging
import threadpool

class CheckZone(object):
    def __init__(self, zone, nodes, serial, start_period):
        self.zone          = zone
        self.master_serial = serial
        self.nodes         = nodes
        self.start_period  = start_period
        self.node_cnt      = len(nodes)
        self.nodes_report  = {}
        logging.debug('INIT:{} = {}'.format(self.zone, self.master_serial))

    def _get_serial(self, nameserver):
        logging.debug('fetching: {}@{}'.format(self.zone, nameserver))
        request = dns.message.make_query(self.zone, dns.rdatatype.SOA)
        try:
            ''' The timeout might need tuning '''
            response = dns.query.udp(request, nameserver, timeout=.25)
        except dns.exception.Timeout:
            logging.error('Timeout from {}@{}'.format(self.zone, nameserver))
            return False
        else:
            return response.answer[0].to_text().split()[6]

    def _check_serial(self, node_dict):
        ''' TODO: Serial number arithmetic? '''
        node = node_dict.items()[0][0]
        addr = node_dict.items()[0][1]
        node_serial = self._get_serial(addr)
        porpagation_time = False
        logging.debug('SERIAL:{}@{} = {}'.format(self.zone, addr, node_serial))
        if node_serial == str(self.master_serial):
            porpagation_time = time.time()
            logging.info('SERIAL:{}@{} == {} @ {}'.format(self.zone, addr, node_serial, 
                porpagation_time))
        elif node_serial > str(self.master_serial):
            logging.debug('SERIAL:{}@{} {} > {} @ {}'.format(self.zone, addr, node_serial, 
                self.master_serial, time.time() ))
            porpagation_time = time.time()
        elif not node_serial:
            porpagation_time = 'Timeout'
        else:
            logging.debug('SERIAL:{}@{} {} != {} @ {}'.format(self.zone, addr, node_serial, 
                self.master_serial, time.time() ))
        return (node, porpagation_time)


    def _thread_callback(self, request, porpagation_time_tuple):
        node = porpagation_time_tuple[0]
        porpagation_time = porpagation_time_tuple[1]
        if porpagation_time:
            self.nodes_report[node] = porpagation_time

    def check_propagation(self):
        ''' Create threadpool before while loop and destroy it after the loop
            or you get a thread leak! '''
        main = threadpool.ThreadPool(self.node_cnt)
        while self.node_cnt != len(self.nodes_report):
            data = [ dict({node : self.nodes[node]}) for node in self.nodes 
                    if node not in self.nodes_report ]
            requests = threadpool.makeRequests(self._check_serial, 
                    data, self._thread_callback)
            [main.putRequest(req) for req in requests]
            main.wait()
            logging.info('processed: {}/{}'.format(len(self.nodes_report), self.node_cnt))
            if self.node_cnt != len(self.nodes_report):
                ''' Report resolution is in seconds so sleep for a bit '''
                time.sleep(0.25)
        main.dismissWorkers(self.node_cnt,True)
        logging.info('{}: processed propagation:'.format(self.zone))
 
def get_enabled_hosts(yaml_dir):
    nodes = {}
    for document in glob.glob(os.path.join(yaml_dir, '*.l.root-servers.org.yaml')):
        node_name = os.path.basename(document).split('.')[0]
        document = file(document, 'r')
        yaml_obj = yaml.load(document)
        logging.debug('YAML:ADD:{}'.format(node_name))
        if not yaml_obj['status']['operational']:
            continue
        for interface, config in yaml_obj['network::interfaces'].items():
            if 'gw4' in config:
                nodes[node_name] = config['addr4'].split('/')[0]
                break
    return nodes


def parse_args():
    parser = argparse.ArgumentParser(description=__doc__)
    parser = argparse.ArgumentParser(description='Preform RSSAC propogation measurment')
    parser.add_argument('--yaml-dir', default='/etc/hierdata/icann-nodes/')
    parser.add_argument('-v', '--verbose', action='count')
    parser.add_argument('zone')
    return parser.parse_args()

def main():
    args      = parse_args()
    log_level = logging.ERROR
    nodes     = get_enabled_hosts(args.yaml_dir)

    if args.verbose == 1:
        log_level = logging.WARN
    elif args.verbose == 2:
        log_level = logging.INFO
    elif args.verbose == 3:
        log_level = logging.DEBUG
    logging.basicConfig(level=log_level)
    



if __name__ == "__main__":
    main()
#!/usr/bin/env python

# 
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

# Original developed by john.bond@icann.org
# Modified for use with Hedgehog by Sinodun IT (www.sinodun.com)

import sys
import time
import rssac
import socket
import argparse
import datetime
import logging
import StringIO
import SocketServer
import dns.name
import dns.tsig
import dns.query
import dns.message
import dns.tsigkeyring
import psycopg2
import os, pwd, grp
import yaml

class DnsReaderServer(SocketServer.UDPServer):
    '''
    SocketServer.ThreadingUDPServer 

    Instance variables:
    
    - RequestHandlerClass
    '''

    keyname   = None
    keyring   = None
    keyalgorithm = None
    
    def __init__(self, server_address, RequestHandlerClass, server_name, zone_name, key=None, keyname=None, keyalgo=None):
        SocketServer.UDPServer.__init__(
                self, server_address, RequestHandlerClass)
        
        config_doc = file('@HEDGEHOGYAML@/hedgehog.yaml', 'r')
        config = yaml.load(config_doc)
        config_doc.close()
        db_host = config['database']['host']
        db_port = config['database']['port']
        db_name = config['database']['name']
        db_owner =  config['database']['owner']
        db_password = config['database']['owner_pass']
                
        ''' We are bound to any restricted ports by this stage '''
        ''' Now do everything as a non-root user '''
        ''' Get the uid/gid from the db_owner name '''
        ''' This assumes that db_owner == username to execute as '''
        running_uid = pwd.getpwnam(db_owner).pw_uid
        running_gid = grp.getgrnam(db_owner).gr_gid
        ''' Clear group privileges '''
        os.setgroups([])
        ''' set new uid/gid '''
        os.setgid(running_gid)
        os.setuid(running_uid)
        
        ''' Start logger '''
        self.logger = logging.getLogger('rssac_propagation.server.DnsReaderServer')
        self.zone_name = zone_name
        
        if key and keyname:
            self.keyname          = keyname
            self.keyring          = dns.tsigkeyring.from_text({ keyname : key})
        if keyalgo:
            self.keyalgorithm     = dns.name.from_text(keyalgo)
            
        ''' Set up database connection '''
        self.conn = psycopg2.connect(database=db_name, user=db_owner, password=db_password,
                                host=db_host, port=db_port)
        cur = self.conn.cursor()
        sql = "select version from dsc.version"
        self.logger.debug('SQL Query: {}'.format(sql))
        cur.execute(sql)
        api_tuple = cur.fetchone()
        if api_tuple == None:
            cur.close()
            self.logger.error('SQL Result: Databae API version not defined in the database')
            raise ValueError('Databae API version not defined in the database')
        if api_tuple[0] != @DB_VERSION@:
            cur.close()
            self.logger.error('SQL Result: Incorrect database API is {} should be {}'.format(api_tuple[0], @DB_VERSION@))
            raise ValueError('Incorrect database API')
        self.logger.debug('SQL Result: Database API is {}'.format(api_tuple[0]))
        
        ''' Get plot id '''
        self.load_time_plot_id = self._get_plot_id('load_time')
        self.zone_size_plot_id = self._get_plot_id('zone_size')

        ''' Get server id '''
        sql = "SELECT id from server where display_name=%s;"
        data = (server_name, )
        fsql = cur.mogrify(sql, data)
        self.logger.debug('SQL Query: {}'.format(fsql))
        cur.execute(sql, data)
        server_id_tuple = cur.fetchone()
        if server_id_tuple == None:
            cur.close()
            raise ValueError('Server is not defined in the database')
        self.logger.debug('SQL Result: Server {} has id {}'.format(server_name, server_id_tuple[0]))
        self.db_server_id = server_id_tuple[0]
        
        ''' Get the list of nodes to query '''
        self.nodes = {}
        sql="SELECT id, ip FROM node WHERE server_id=%s and ip is not null"
        data = (self.db_server_id, )
        fsql = cur.mogrify(sql, data)
        self.logger.debug('SQL: {}'.format(fsql))
        cur.execute(sql, data)
        for node in cur:
            self.nodes[node[0]] = node[1]
        if not self.nodes:
            cur.close()
            raise ValueError('No nodes found for server id {} with an IP address'.format(self.db_server_id))
        self.logger.debug('SQL Result: Server id {} has IP Adresses for node(s) id {}'.format(self.db_server_id, self.nodes))
    
        ''' Get max serial number '''
        ''' This is read at startup and state is maintained while the server is running '''
        sql = "SELECT max(data.key2) FROM data WHERE plot_id=%s AND server_id=%s"
        data = (self.load_time_plot_id, self.db_server_id)
        fsql = cur.mogrify(sql, data)
        self.logger.debug('SQL: {}'.format(fsql))
        cur.execute(sql, data)
        max_serial_tuple = cur.fetchone()
        if max_serial_tuple[0] == None:
            self.max_serial=0
        else:
            self.max_serial=max_serial_tuple[0]
    
        cur.close()
        
    def _get_plot_id(self, plot_name):
        cur = self.conn.cursor()
        sql = "SELECT id from dataset where name=%s;"
        data = (plot_name, )
        fsql = cur.mogrify(sql, data)
        self.logger.debug('SQL Query: {}'.format(fsql))
        cur.execute(sql, data)
        plot_id_tuple = cur.fetchone()
        if plot_id_tuple == None:
            cur.close()
            raise ValueError('Plot is not defined in the database')
        self.logger.debug('SQL Result: Plot {} has id {}'.format(plot_name, plot_id_tuple[0]))
        cur.close()
        return plot_id_tuple[0]
        
    
        
class DnsReaderHandler(SocketServer.BaseRequestHandler):
    '''
    Base Handler class 
    '''

    message  = None
    serial   = None
    data     = None
    socket   = None
    qname    = None

    def __init__(self, request, client_address, server):
        SocketServer.BaseRequestHandler.__init__(self, request, client_address, server)

    def _process_load_time(self):
        start = time.time()
        self.server.logger.debug('SQL Result: Server id {} has max serial {}'.format(self.server.db_server_id, self.server.max_serial))
        zone_check = rssac.CheckZone(self.qname, self.server.nodes,
                self.serial, start)
        zone_check.check_propagation()
        cur = self.server.conn.cursor()
        for node, end in zone_check.nodes_report.items():
            if type(end) is str:
                ''' Timeout '''
                load_time = None
                continue
            
            load_time = end - start
            sql = "INSERT INTO data (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (%s,%s,%s,%s,%s,%s,%s)"
            data = (datetime.datetime.fromtimestamp(start), self.server.db_server_id, node, self.server.load_time_plot_id, self.qname, self.serial, load_time)
            fsql = cur.mogrify(sql, data)
            self.server.logger.debug('SQL: {}'.format(fsql))
            cur.execute(sql, data)
        
        '''' record this serial as the new max(serial) '''
        self.server.max_serial = self.serial
        self.server.conn.commit()
        cur.close()
        
    def _get_zone_size(self):
        self.server.logger.debug('Starting get_zone_size')
        zone      = StringIO.StringIO()
        zone_size = 0
        self.server.logger.debug('xfr query address {} qname {} keyname {}'.format(self.client_address[0], self.qname, self.server.keyname))
        xfr       = dns.query.xfr(self.client_address[0], self.qname, keyname=self.server.keyname,
                    keyring=self.server.keyring, keyalgorithm=self.server.keyalgorithm)
        try:
            for message in xfr:
                for ans in message.answer:
                    ans.to_wire(zone, origin=dns.name.root)
        except dns.exception.FormError:
            self.server.logger.error('Error attempting AXFR from {}'.format(self.client_address[0]))
        else:
            ''' Need to subtract 2 bytes for the TCP length field '''
            zone_size = sys.getsizeof(zone.getvalue()) - 2
        zone.close()
        self.server.logger.debug('Done get_zone_size')
        return zone_size

    def _process_zone_size(self):
        self.server.logger.debug('Starting process_zone_size')
        start = time.time()
        zone_size = self._get_zone_size()
        self.server.logger.debug('zone_size {}'.format(zone_size))
        if zone_size == 0:
            ''' Return because we didn't get an answer '''
            self.server.logger.warn('Warning: No zone size returned from {}'.format(self.client_address[0]))
            return
        cur = self.server.conn.cursor()
        for node in self.server.nodes.keys():
            sql = "INSERT INTO data (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (%s,%s,%s,%s,%s,%s,%s)"
            data = (datetime.datetime.fromtimestamp(start), self.server.db_server_id, node, self.server.zone_size_plot_id, self.qname, self.serial, zone_size)
            fsql = cur.mogrify(sql, data)
            self.server.logger.debug('SQL: {}'.format(fsql))
            cur.execute(sql, data)
            self.server.conn.commit()
        cur.close()    
        self.server.logger.debug('Done process_zone_size')
        
    def send_response(self):
        ''' Send notify response '''
        response = dns.message.make_response(self.message)
        self.socket.sendto(response.to_wire(), self.client_address)
        
    def parse_dns(self):
        '''
        parse the data package into dns elements
        '''
        self.data = str(self.request[0]).strip()
        self.socket = self.request[1]
        #incoming Data
        try:
            self.message = dns.message.from_wire(self.data)
        except dns.name.BadLabelType:
            #Error processing label (bit flip?)
            self.server.logger.error('Received Bad label Type from {}'.format(self.client_address[0]))
        except dns.message.ShortHeader:
            #Received junk
            self.server.logger.error('Received Junk from {}'.format(self.client_address[0]))
        else:
            current_time = int(time.time())
            if self.message.opcode() == 4:
                self.qname = self.message.question[0].name.to_text()
                if self.qname == self.server.zone_name:
                    self.server.logger.debug('Received notify for zone {}'.format(self.qname))
                    if len(self.message.answer) > 0:
                        answer = self.message.answer[0]
                        self.serial = answer.to_rdataset()[0].serial
                    
                        if int(self.serial) <= int(self.server.max_serial):
                            self.server.logger.debug('{}:{}:load-time already processed or lower then max({})'.format(
                                self.qname, self.serial, self.server.max_serial))
                            self.send_response()
                            return False
                        else:    
                            self.server.logger.debug('Received notify for {} from {}'.format(self.serial, self.client_address[0]))
                            self.send_response()
                            return True
                    else:
                        self.server.logger.error('Received notify with no serial from {}'.format(self.client_address[0]))
                        self.send_response()
                else:
                    self.server.logger.debug('Received notify for zone {} expected zone {}, ignoring'.format(self.server.zone_name, self.qname))
        return False

    def handle(self):
        '''
        RequestHandlerClass handle function
        handler listens for dns packets
        '''
        if self.parse_dns():
            self._process_load_time()
            self._process_zone_size()
    
def main():
    ''' parse cmd line args '''
    parser = argparse.ArgumentParser(description='nofify receiver')
    parser.add_argument('-n','--tsig-name')
    parser.add_argument('-k','--tsig-key')
    parser.add_argument('-a','--tsig-algo', 
            choices=['hmac-sha1', 'hmac-sha224', 'hmac-sha256', 'hmac-sha384','hmac-sha512'])
    parser.add_argument('-v', '--verbose', action='count', help='increase verbosity (up to -vvv)')
    parser.add_argument('-o','--log', default='server.log')
    parser.add_argument('-l', '--listen', metavar="0.0.0.0:53", 
            default="0.0.0.0:53", help='listen on address:port ')
    parser.add_argument('-s','--server', help='server name', required=True)
    parser.add_argument('-z','--zone',help='zone name', required=True)
    args = parser.parse_args()
    host, port = args.listen.split(":")
    
    ''' configure logging '''
    log_level = logging.ERROR
    log_format ='%(asctime)s:%(levelname)s:%(name)s:%(funcName)s(%(levelno)s):%(message)s'
    if args.verbose == 1:
        log_level = logging.WARN
    elif args.verbose == 2:
        log_level = logging.INFO
    elif args.verbose > 2:
        log_level = logging.DEBUG
    logging.basicConfig(level=log_level, filename=args.log, format=log_format)
    logger = logging.getLogger('rssac_propagation.server')
    logger.info('rssacd starting up...')
    
    ''' Init and then run the server '''
    server = DnsReaderServer((host, int(port)), DnsReaderHandler, args.server, args.zone,
            args.tsig_key, args.tsig_name, args.tsig_algo) 
    server.serve_forever()
    
if __name__ == "__main__":
    main()
TSIG_NAME=""
TSIG_KEY=""
# One of hmac-sha1,hmac-sha224,hmac-sha256,hmac-sha384,hmac-sha512
TSIG_ALGO=""
LOG="/var/log/hedgehog/rssacd.log"
LISTEN="0.0.0.0:53"
# Server name 
SERVER="Server-A"
ZONE="."
#OTHER_ARGS="-v"
#!@PYTHON_EXE@

# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

#
# Original developed by john.bond@icann.org
# Modified for use with Hedgehog by Sinodun IT (www.sinodun.com)

import sys
sys.path.append('@HHPYSITEDIR@')
from rssac_server import main
main()
description "RSSAC02 Monitor"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 10 5
umask 022
script 
 . @HEDGEHOGYAML@/rssacd.conf
 exec @SBIN@/rssacd -n $TSIG_NAME -k $TSIG_KEY -a $TSIG_ALGO -o $LOG --listen $LISTEN --server $SERVER --zone $ZONE $OTHER_ARGS
end script
# -*- coding: UTF-8 -*-
"""Easy to use object-oriented thread pool framework.

A thread pool is an object that maintains a pool of worker threads to perform
time consuming operations in parallel. It assigns jobs to the threads
by putting them in a work request queue, where they are picked up by the
next available thread. This then performs the requested operation in the
background and puts the results in another queue.

The thread pool object can then collect the results from all threads from
this queue as soon as they become available or after all threads have
finished their work. It's also possible, to define callbacks to handle
each result as it comes in.

The basic concept and some code was taken from the book "Python in a Nutshell,
2nd edition" by Alex Martelli, O'Reilly 2006, ISBN 0-596-10046-9, from section
14.5 "Threaded Program Architecture". I wrapped the main program logic in the
ThreadPool class, added the WorkRequest class and the callback system and
tweaked the code here and there. Kudos also to Florent Aide for the exception
handling mechanism.

Basic usage::

    >>> pool = ThreadPool(poolsize)
    >>> requests = makeRequests(some_callable, list_of_args, callback)
    >>> [pool.putRequest(req) for req in requests]
    >>> pool.wait()

See the end of the module code for a brief, annotated usage example.

Website : http://chrisarndt.de/projects/threadpool/

"""
__docformat__ = "restructuredtext en"

__all__ = [
    'makeRequests',
    'NoResultsPending',
    'NoWorkersAvailable',
    'ThreadPool',
    'WorkRequest',
    'WorkerThread'
]

__author__ = "Christopher Arndt"
__version__ = '1.2.7'
__revision__ = "$Revision: 416 $"
__date__ = "$Date: 2009-10-07 05:41:27 +0200 (Wed, 07 Oct 2009) $"
__license__ = "MIT license"


# standard library modules
import sys
import threading
import Queue
import traceback


# exceptions
class NoResultsPending(Exception):
    """All work requests have been processed."""
    pass

class NoWorkersAvailable(Exception):
    """No worker threads available to process remaining requests."""
    pass


# internal module helper functions
def _handle_thread_exception(request, exc_info):
    """Default exception handler callback function.

    This just prints the exception info via ``traceback.print_exception``.

    """
    traceback.print_exception(*exc_info)


# utility functions
def makeRequests(callable_, args_list, callback=None,
        exc_callback=_handle_thread_exception):
    """Create several work requests for same callable with different arguments.

    Convenience function for creating several work requests for the same
    callable where each invocation of the callable receives different values
    for its arguments.

    ``args_list`` contains the parameters for each invocation of callable.
    Each item in ``args_list`` should be either a 2-item tuple of the list of
    positional arguments and a dictionary of keyword arguments or a single,
    non-tuple argument.

    See docstring for ``WorkRequest`` for info on ``callback`` and
    ``exc_callback``.

    """
    requests = []
    for item in args_list:
        if isinstance(item, tuple):
            requests.append(
                WorkRequest(callable_, item[0], item[1], callback=callback,
                    exc_callback=exc_callback)
            )
        else:
            requests.append(
                WorkRequest(callable_, [item], None, callback=callback,
                    exc_callback=exc_callback)
            )
    return requests


# classes
class WorkerThread(threading.Thread):
    """Background thread connected to the requests/results queues.

    A worker thread sits in the background and picks up work requests from
    one queue and puts the results in another until it is dismissed.

    """

    def __init__(self, requests_queue, results_queue, poll_timeout=5, **kwds):
        """Set up thread in daemonic mode and start it immediatedly.

        ``requests_queue`` and ``results_queue`` are instances of
        ``Queue.Queue`` passed by the ``ThreadPool`` class when it creates a new
        worker thread.

        """
        threading.Thread.__init__(self, **kwds)
        self.setDaemon(1)
        self._requests_queue = requests_queue
        self._results_queue = results_queue
        self._poll_timeout = poll_timeout
        self._dismissed = threading.Event()
        self.start()

    def run(self):
        """Repeatedly process the job queue until told to exit."""
        while True:
            if self._dismissed.isSet():
                # we are dismissed, break out of loop
                break
            # get next work request. If we don't get a new request from the
            # queue after self._poll_timout seconds, we jump to the start of
            # the while loop again, to give the thread a chance to exit.
            try:
                request = self._requests_queue.get(True, self._poll_timeout)
            except Queue.Empty:
                continue
            else:
                if self._dismissed.isSet():
                    # we are dismissed, put back request in queue and exit loop
                    self._requests_queue.put(request)
                    break
                try:
                    result = request.callable(*request.args, **request.kwds)
                    self._results_queue.put((request, result))
                except:
                    request.exception = True
                    self._results_queue.put((request, sys.exc_info()))

    def dismiss(self):
        """Sets a flag to tell the thread to exit when done with current job."""
        self._dismissed.set()


class WorkRequest:
    """A request to execute a callable for putting in the request queue later.

    See the module function ``makeRequests`` for the common case
    where you want to build several ``WorkRequest`` objects for the same
    callable but with different arguments for each call.

    """

    def __init__(self, callable_, args=None, kwds=None, requestID=None,
            callback=None, exc_callback=_handle_thread_exception):
        """Create a work request for a callable and attach callbacks.

        A work request consists of the a callable to be executed by a
        worker thread, a list of positional arguments, a dictionary
        of keyword arguments.

        A ``callback`` function can be specified, that is called when the
        results of the request are picked up from the result queue. It must
        accept two anonymous arguments, the ``WorkRequest`` object and the
        results of the callable, in that order. If you want to pass additional
        information to the callback, just stick it on the request object.

        You can also give custom callback for when an exception occurs with
        the ``exc_callback`` keyword parameter. It should also accept two
        anonymous arguments, the ``WorkRequest`` and a tuple with the exception
        details as returned by ``sys.exc_info()``. The default implementation
        of this callback just prints the exception info via
        ``traceback.print_exception``. If you want no exception handler
        callback, just pass in ``None``.

        ``requestID``, if given, must be hashable since it is used by
        ``ThreadPool`` object to store the results of that work request in a
        dictionary. It defaults to the return value of ``id(self)``.

        """
        if requestID is None:
            self.requestID = id(self)
        else:
            try:
                self.requestID = hash(requestID)
            except TypeError:
                raise TypeError("requestID must be hashable.")
        self.exception = False
        self.callback = callback
        self.exc_callback = exc_callback
        self.callable = callable_
        self.args = args or []
        self.kwds = kwds or {}

    def __str__(self):
        return "<WorkRequest id=%s args=%r kwargs=%r exception=%s>" % \
            (self.requestID, self.args, self.kwds, self.exception)

class ThreadPool:
    """A thread pool, distributing work requests and collecting results.

    See the module docstring for more information.

    """

    def __init__(self, num_workers, q_size=0, resq_size=0, poll_timeout=5):
        """Set up the thread pool and start num_workers worker threads.

        ``num_workers`` is the number of worker threads to start initially.

        If ``q_size > 0`` the size of the work *request queue* is limited and
        the thread pool blocks when the queue is full and it tries to put
        more work requests in it (see ``putRequest`` method), unless you also
        use a positive ``timeout`` value for ``putRequest``.

        If ``resq_size > 0`` the size of the *results queue* is limited and the
        worker threads will block when the queue is full and they try to put
        new results in it.

        .. warning:
            If you set both ``q_size`` and ``resq_size`` to ``!= 0`` there is
            the possibilty of a deadlock, when the results queue is not pulled
            regularly and too many jobs are put in the work requests queue.
            To prevent this, always set ``timeout > 0`` when calling
            ``ThreadPool.putRequest()`` and catch ``Queue.Full`` exceptions.

        """
        self._requests_queue = Queue.Queue(q_size)
        self._results_queue = Queue.Queue(resq_size)
        self.workers = []
        self.dismissedWorkers = []
        self.workRequests = {}
        self.createWorkers(num_workers, poll_timeout)

    def createWorkers(self, num_workers, poll_timeout=5):
        """Add num_workers worker threads to the pool.

        ``poll_timout`` sets the interval in seconds (int or float) for how
        ofte threads should check whether they are dismissed, while waiting for
        requests.

        """
        for i in range(num_workers):
            self.workers.append(WorkerThread(self._requests_queue,
                self._results_queue, poll_timeout=poll_timeout))

    def dismissWorkers(self, num_workers, do_join=False):
        """Tell num_workers worker threads to quit after their current task."""
        dismiss_list = []
        for i in range(min(num_workers, len(self.workers))):
            worker = self.workers.pop()
            worker.dismiss()
            dismiss_list.append(worker)

        if do_join:
            for worker in dismiss_list:
                worker.join()
        else:
            self.dismissedWorkers.extend(dismiss_list)

    def joinAllDismissedWorkers(self):
        """Perform Thread.join() on all worker threads that have been dismissed.
        """
        for worker in self.dismissedWorkers:
            worker.join()
        self.dismissedWorkers = []

    def putRequest(self, request, block=True, timeout=None):
        """Put work request into work queue and save its id for later."""
        assert isinstance(request, WorkRequest)
        # don't reuse old work requests
        assert not getattr(request, 'exception', None)
        self._requests_queue.put(request, block, timeout)
        self.workRequests[request.requestID] = request

    def poll(self, block=False):
        """Process any new results in the queue."""
        while True:
            # still results pending?
            if not self.workRequests:
                raise NoResultsPending
            # are there still workers to process remaining requests?
            elif block and not self.workers:
                raise NoWorkersAvailable
            try:
                # get back next results
                request, result = self._results_queue.get(block=block)
                # has an exception occured?
                if request.exception and request.exc_callback:
                    request.exc_callback(request, result)
                # hand results to callback, if any
                if request.callback and not \
                       (request.exception and request.exc_callback):
                    request.callback(request, result)
                del self.workRequests[request.requestID]
            except Queue.Empty:
                break

    def wait(self):
        """Wait for results, blocking until all have arrived."""
        while 1:
            try:
                self.poll(True)
            except NoResultsPending:
                break


################
# USAGE EXAMPLE
################

if __name__ == '__main__':
    import random
    import time

    # the work the threads will have to do (rather trivial in our example)
    def do_something(data):
        time.sleep(random.randint(1,5))
        result = round(random.random() * data, 5)
        # just to show off, we throw an exception once in a while
        if result > 5:
            raise RuntimeError("Something extraordinary happened!")
        return result

    # this will be called each time a result is available
    def print_result(request, result):
        print "**** Result from request #%s: %r" % (request.requestID, result)

    # this will be called when an exception occurs within a thread
    # this example exception handler does little more than the default handler
    def handle_exception(request, exc_info):
        if not isinstance(exc_info, tuple):
            # Something is seriously wrong...
            print request
            print exc_info
            raise SystemExit
        print "**** Exception occured in request #%s: %s" % \
          (request.requestID, exc_info)

    # assemble the arguments for each job to a list...
    data = [random.randint(1,10) for i in range(20)]
    # ... and build a WorkRequest object for each item in data
    requests = makeRequests(do_something, data, print_result, handle_exception)
    # to use the default exception handler, uncomment next line and comment out
    # the preceding one.
    #requests = makeRequests(do_something, data, print_result)

    # or the other form of args_lists accepted by makeRequests: ((,), {})
    data = [((random.randint(1,10),), {}) for i in range(20)]
    requests.extend(
        makeRequests(do_something, data, print_result, handle_exception)
        #makeRequests(do_something, data, print_result)
        # to use the default exception handler, uncomment next line and comment
        # out the preceding one.
    )

    # we create a pool of 3 worker threads
    print "Creating thread pool with 3 worker threads."
    main = ThreadPool(3)

    # then we put the work requests in the queue...
    for req in requests:
        main.putRequest(req)
        print "Work request #%s added." % req.requestID
    # or shorter:
    # [main.putRequest(req) for req in requests]

    # ...and wait for the results to arrive in the result queue
    # by using ThreadPool.wait(). This would block until results for
    # all work requests have arrived:
    # main.wait()

    # instead we can poll for results while doing something else:
    i = 0
    while True:
        try:
            time.sleep(0.5)
            main.poll()
            print "Main thread working...",
            print "(active worker threads: %i)" % (threading.activeCount()-1, )
            if i == 10:
                print "**** Adding 3 more worker threads..."
                main.createWorkers(3)
            if i == 20:
                print "**** Dismissing 2 worker threads..."
                main.dismissWorkers(2)
            i += 1
        except KeyboardInterrupt:
            print "**** Interrupted!"
            break
        except NoResultsPending:
            print "**** No pending results."
            break
    if main.dismissedWorkers:
        print "Joining all dismissed worker threads..."
        main.joinAllDismissedWorkers()
CLEANFILES = $(bin_SCRIPTS)

bin_SCRIPTS = refile_and_grok \
              hedgehog/hedgehog_database_create \
              hedgehogctl

nobase_libexec_SCRIPTS = hedgehog/conf_read \
                         hedgehog/check_user_perms \
                         hedgehog/database_init \
                         hedgehog/database_manage_partitions \
                         hedgehog/database_update_geoip \
                         hedgehog/database_update_nodes \
                         hedgehog/database_update_service_addr \
                         hedgehog/database_update_tlds \
                         hedgehog/database_update_tlds_from_zone \
                         hedgehog/datafiles_create_summary \
                         hedgehog/datafiles_rm_empty_xml_dirs \
                         hedgehog/datafiles_tar_old_xml_files \
                         hedgehog/plotcache_generate_cached_plots \
                         hedgehog/plotcache_rm_cached_plots \
                         hedgehog/rssac_generate_reports \
                         hedgehog/database_process_rssac_data \
                         hedgehog/database_rm_old_rssac_data \
                         hedgehog/DDL_updates/000010_ddl_python \
                         hedgehog/DDL_updates/000011_ddl_new_graphs \
                         hedgehog/DDL_updates/000012_ddl_lower_key_index \
                         hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn \
                         hedgehog/DDL_updates/000014_ddl_service_addr \
                         hedgehog/DDL_updates/000015_ddl_byregion \
                         hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans \
                         hedgehog/DDL_updates/000017_ddl_node_stats \
                         hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX \
                         hedgehog/utility_prepare_GeoLite2-City-Locations-en

nobase_data_DATA = $(srcdir)/hedgehog/sql/initialise_database.sql \
                   $(srcdir)/hedgehog/sql/insert_lookup_data.sql \
                   $(srcdir)/hedgehog/sql/dns.sql

subst_LIBEXECDATA = sed -e "s|@LIBEXEC@|@libexecdir@/hedgehog|" -e "s|@DATA@|@datadir@|"
subst_DB_VERSION = sed -e "s|@DB_VERSION@|@DB_SCHEMA_VERSION@|"
subst_BIN = sed -e "s|@BIN@|@bindir@|"
do_subst_python = sed -e 's|@PYTHON_EXE@|$(PYTHON)|' -e 's|@HHPYSITEDIR@|$(pythondir)|'
subst_sql = sed -e 's|<SQL_PATH>|$(datadir)/hedgehog/sql|'
subst_version = sed -e 's|<VERSION>|@VERSION@|'

hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX: $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in Makefile
	$(do_subst_python) < $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX.in > hedgehog/utility_prepare_GeoLite2-City-Blocks-IPvX

hedgehog/utility_prepare_GeoLite2-City-Locations-en: $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in Makefile
	$(do_subst_python) < $(srcdir)/hedgehog/utility_prepare_GeoLite2-City-Locations-en.in > hedgehog/utility_prepare_GeoLite2-City-Locations-en
	
refile_and_grok: refile_and_grok.in Makefile
	$(subst_LIBEXECDATA) < $(srcdir)/refile_and_grok.in | $(subst_BIN) > refile_and_grok

hedgehogctl: hedgehogctl.in Makefile
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehogctl.in > hedgehogctl

hedgehog/conf_read: $(srcdir)/hedgehog/conf_read.in Makefile
	mkdir -p hedgehog
	$(subst_HH_CONF_FILE) < $(srcdir)/hedgehog/conf_read.in > hedgehog/conf_read

hedgehog/check_user_perms: $(srcdir)/hedgehog/check_user_perms.in Makefile
	mkdir -p hedgehog
	$(subst_DB_VERSION) < $(srcdir)/hedgehog/check_user_perms.in | $(subst_LIBEXECDATA) | $(subst_version) > hedgehog/check_user_perms
	
hedgehog/database_init: $(srcdir)/hedgehog/database_init.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_init.in > hedgehog/database_init

hedgehog/database_manage_partitions: $(srcdir)/hedgehog/database_manage_partitions.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_manage_partitions.in > hedgehog/database_manage_partitions

hedgehog/database_update_geoip: $(srcdir)/hedgehog/database_update_geoip.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_geoip.in > hedgehog/database_update_geoip

hedgehog/database_update_nodes: $(srcdir)/hedgehog/database_update_nodes.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_nodes.in > hedgehog/database_update_nodes

hedgehog/database_update_service_addr: $(srcdir)/hedgehog/database_update_service_addr.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_service_addr.in > hedgehog/database_update_service_addr

hedgehog/database_update_tlds: $(srcdir)/hedgehog/database_update_tlds.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_tlds.in > hedgehog/database_update_tlds

hedgehog/database_update_tlds_from_zone: $(srcdir)/hedgehog/database_update_tlds_from_zone.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_update_tlds_from_zone.in > hedgehog/database_update_tlds_from_zone

hedgehog/datafiles_create_summary: $(srcdir)/hedgehog/datafiles_create_summary.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/datafiles_create_summary.in > hedgehog/datafiles_create_summary

hedgehog/datafiles_rm_empty_xml_dirs: $(srcdir)/hedgehog/datafiles_rm_empty_xml_dirs.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/datafiles_rm_empty_xml_dirs.in > hedgehog/datafiles_rm_empty_xml_dirs

hedgehog/datafiles_tar_old_xml_files: $(srcdir)/hedgehog/datafiles_tar_old_xml_files.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/datafiles_tar_old_xml_files.in > hedgehog/datafiles_tar_old_xml_files

hedgehog/plotcache_generate_cached_plots: $(srcdir)/hedgehog/plotcache_generate_cached_plots.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/plotcache_generate_cached_plots.in > hedgehog/plotcache_generate_cached_plots

hedgehog/plotcache_rm_cached_plots: $(srcdir)/hedgehog/plotcache_rm_cached_plots.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/plotcache_rm_cached_plots.in > hedgehog/plotcache_rm_cached_plots

hedgehog/rssac_generate_reports: $(srcdir)/hedgehog/rssac_generate_reports.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/rssac_generate_reports.in > hedgehog/rssac_generate_reports

hedgehog/database_process_rssac_data: $(srcdir)/hedgehog/database_process_rssac_data.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_process_rssac_data.in > hedgehog/database_process_rssac_data

hedgehog/database_rm_old_rssac_data: $(srcdir)/hedgehog/database_rm_old_rssac_data.in Makefile
	mkdir -p hedgehog
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/database_rm_old_rssac_data.in > hedgehog/database_rm_old_rssac_data

hedgehog/DDL_updates/000010_ddl_python: $(srcdir)/hedgehog/DDL_updates/000010_ddl_python.in Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000010_ddl_python.in | $(subst_DB_VERSION) | $(subst_sql) > hedgehog/DDL_updates/000010_ddl_python

hedgehog/DDL_updates/000011_ddl_new_graphs: $(srcdir)/hedgehog/DDL_updates/000011_ddl_new_graphs.in Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000011_ddl_new_graphs.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000011_ddl_new_graphs

hedgehog/DDL_updates/000012_ddl_lower_key_index: $(srcdir)/hedgehog/DDL_updates/000012_ddl_lower_key_index.in Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000012_ddl_lower_key_index.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000012_ddl_lower_key_index

hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn: $(srcdir)/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn.in  | $(subst_DB_VERSION) > hedgehog/DDL_updates/000013_ddl_dnskey_vs_asn

hedgehog/DDL_updates/000014_ddl_service_addr: $(srcdir)/hedgehog/DDL_updates/000014_ddl_service_addr.in  Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000014_ddl_service_addr.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000014_ddl_service_addr

hedgehog/DDL_updates/000015_ddl_byregion: $(srcdir)/hedgehog/DDL_updates/000015_ddl_byregion.in Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000015_ddl_byregion.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000015_ddl_byregion

hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans: $(srcdir)/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans.in | $(subst_DB_VERSION) > hedgehog/DDL_updates/000016_ddl_server_addr_vs_trans

hedgehog/DDL_updates/000017_ddl_node_stats: $(srcdir)/hedgehog/DDL_updates/000017_ddl_node_stats.in Makefile
	mkdir -p hedgehog/DDL_updates
	$(subst_LIBEXECDATA) < $(srcdir)/hedgehog/DDL_updates/000017_ddl_node_stats.in | $(subst_DB_VERSION) >hedgehog/DDL_updates/000017_ddl_node_stats

EXTRA_DIST = refile_and_grok.in \
             conf_read.in \
             database_init.in \
             database_manage_partitions.in \
             database_update_nodes.in \
             datafiles_create_summary.in \
             datafiles_rm_empty_xml_dirs.in \
             datafiles_tar_old_xml_files.in \
             plotcache_generate_cached_plots.in \
             plotcache_rm_cached_plots.in \
             rssac_generate_reports.in \
             database_process_rssac_data.in \
             database_rm_old_rssac_data.in 

install-data-hook:
	mkdir -p ${DESTDIR}$(localstatedir)/hedgehog/data
	chmod -R 775 ${DESTDIR}$(localstatedir)/hedgehog/data


dist_man_MANS = hedgehogctl.1 \
                refile_and_grok.1 \
                hedgehog_database_create.1

hedgehogctl.1: hedgehogctl Makefile
	chmod 755 hedgehogctl
	$(HELP2MAN) -h help2man --version-string=@PACKAGE_VERSION@ -N -s 1 -n "Wrapper around hedgehog management scripts" ./hedgehogctl -o hedgehogctl.1
	
refile_and_grok.1: refile_and_grok Makefile
	chmod 755 refile_and_grok
	$(HELP2MAN) -h -h --version-string=@PACKAGE_VERSION@ -N -s 1 -n "Wrapper around the dsc-extractor application that launches multiple extractors to process data" ./refile_and_grok -o refile_and_grok.1

hedgehog_database_create.1: $(srcdir)/hedgehog/hedgehog_database_create Makefile
	$(HELP2MAN) -h -h --version-string=@PACKAGE_VERSION@ -N -s 1 -n "This script is to aid the database administrator with creating the hedgehog database." $(srcdir)/hedgehog/hedgehog_database_create -o hedgehog_database_create.1
#!/bin/bash
#
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000010_ddl_python
#

eval $(@LIBEXEC@/conf_read)

DB_NAME=$hhconfig_database_name
DATA_DIR=$hhconfig_directories_data
SQ="'"

NEW_DDL_SERIAL=10
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=8
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))

DSC_SCHEMA="dsc"

usage () {
  echo
  echo "Usage: $(basename $0) options"
  echo
  echo "Supported options:"
  echo "  -d Run in debug mode"
  echo "  -h Show this help."
  echo "  -P <password for the postgresql super user>"
  echo "  -U <username for the postgresql super user> Required"
  echo "  -v Show version"
  exit 1
}

while getopts ":hP:U:v" opt; do
  case $opt in
    d  ) set -x ;;
    h  ) usage ;;
    P  ) SPECIFIED_PASS=$OPTARG ;;
    U  ) SPECIFIED_USER=$OPTARG ;;
    v  ) echo "<VERSION>"
         exit 0 ;;
    \? ) usage ;;
  esac
done

if [ -z "$SPECIFIED_USER" ] ; then
  usage
fi

# We need a superuser. 
a=$(@LIBEXEC@/check_user_perms -U $SPECIFIED_USER -P $SPECIFIED_PASS -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
if [ $? -ne 0 ] ; then
  echo $a
  exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

echo "$(date): $0 COMMENCED"

psql $CONN_STRING -qtc '\dL' | grep plpythonu
if [ $? -eq 1 ] ; then
  echo "No plpythonu language installed. Ensure that the plpythonu package is installed."
  echo "Then connect to the $DB_NAME database as a superuser and run the command \"CREATE EXTENSION plpythonu SCHEMA pg_catalog;\""
  exit 1
fi

psql $CONN_STRING < <SQL_PATH>/dns.sql
if [ $? -eq 0 ] ; then
psql $CONN_STRING <<EOF
BEGIN;

UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new ip2asn functions', now() );
COMMIT;
EOF
fi

echo "`date`: $0 COMPLETED"
#!/bin/bash
#
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000011_ddl_new_graphs
#

eval $(@LIBEXEC@/conf_read)

DB_NAME=$hhconfig_database_name
DATA_DIR=$hhconfig_directories_data
SQ="'"
export PGPASSWORD=$hhconfig_database_owner_pass

NEW_DDL_SERIAL=11
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=9
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))

DSC_SCHEMA="dsc"

usage () {
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
    exit 1
}


while getopts ":csh" opt; do
    case $opt in
        h  ) usage ;;
        \? ) usage ;;
    esac
done

# Need a write user to update DB
echo "`date`: $0 COMMENCED"
a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
if [ $? -ne 0 ] ; then
  echo $a
  exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

echo "Starting"

# START TLD
TLD_DATASET=$(psql -h $hhconfig_database_host $DB_NAME -qtc "SELECT id FROM dataset WHERE name = 'qtype_vs_tld';")
# END  TLD
# START GEOIP
psql -h $hhconfig_database_host $DB_NAME -qtc '\dx' | grep ip4r | grep dsc
[ $? -eq 1 ] && \
echo "No ip4r extension installed. Ensure that the ip4r package is installed." && \
echo "Then connect to the $DB_NAME database as a superuser and run the command \"CREATE EXTENSION ip4r SCHEMA dsc;\"" && \
exit 1
GEO_DATASET=$(psql -h $hhconfig_database_host $DB_NAME -qtc "SELECT id FROM dataset WHERE name = 'client_subnet_accum';")
# END GEOIP
# START IP2ASN
IP2ASN_DATASET=$(psql $CONN_STRING -qtc "SELECT id FROM dataset WHERE name = 'client_subnet_accum';")
IP2ASN_DATASET2=$(psql $CONN_STRING -qtc "SELECT id FROM dataset WHERE name = 'client_addr_vs_rcode_accum';")
IP2ASN_DATASET3=$(psql $CONN_STRING -qtc "SELECT id FROM dataset WHERE name = 'client_subnet2_accum';")
# END IP2ASN

echo "Adding graphs"


psql -h $hhconfig_database_host $DB_NAME <<EOF
BEGIN;
-- START TLDs
CREATE TABLE tld_types (id serial PRIMARY KEY, type_name text);
INSERT INTO tld_types (type_name) VALUES
  ('New-gTLD'),
  ('Legacy-gTLD'),
  ('ccTLD');
CREATE TABLE tlds (id serial PRIMARY KEY,
                   alabel text UNIQUE,
                   type int REFERENCES tld_types (id));

-- INSERT INTO dataset (name) VALUES ('# client_subnet_vs_tld');

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('qtype_vs_cctld','3.1.QTYPE vs TLD','QTYPE for most popular ccTLDs','QTYPE value for most popular ccTLDs queried') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${TLD_DATASET} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title)
    VALUES ('qtype_vs_newgtld','3.1.QTYPE vs TLD','QTYPE for most popular New gTLDs','QTYPE value for most popular New gTLDs queried') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${TLD_DATASET} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('qtype_vs_legacygtld','3.1.QTYPE vs TLD','QTYPE for most popular Legacy gTLDs','QTYPE value for most popular Legacy gTLDs queried') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot select id, ${TLD_DATASET} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('qtype_vs_othertld','3.1.QTYPE vs TLD','QTYPE for most popular Undelegated TLDs','QTYPE value for most popular Undelegated TLDs queried') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${TLD_DATASET} FROM rows;

/* WITH rows AS (
*   INSERT INTO visible_plot (name, display_category, display_name, title)
*     VALUES ('client_subnet_vs_tld','7.Client Subnet Statistics','Busiest client subnet vs TLD','Client subnet vs TLD')
*     RETURNING id
* )
* INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='client_subnet_vs_tld') FROM rows;
*/

UPDATE visible_plot set display_category = '3.1.QTYPE vs TLD' where name='qtype_vs_tld';
UPDATE visible_plot set display_name='RCODE by client subnet (fixed size)' where name='client_addr_vs_rcode_accum';
-- END TLDs

-- START GEO
DROP TABLE IF EXISTS geo;
DROP SEQUENCE IF EXISTS geo_id_seq;

CREATE TABLE geoip (
  ip_range iprange PRIMARY KEY,
  latitude real,
  longitude real,
  geoname integer );

CREATE INDEX ix_ip_range ON geoip USING gist (ip_range);

CREATE TABLE locations (
  geoname_id integer PRIMARY KEY,
  name text,
  city text );

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('geomap','10.GEO','Busiest client locations by city','Busiest client locations by city') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${GEO_DATASET} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('geochart','10.GEO','Busiest client locations by country','Busiest client locations by country, node locations by city') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${GEO_DATASET} FROM rows;
-- END GEO

-- START NODE GROUPING
ALTER TABLE node  ADD column subgroup character varying(255);

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('by_instance','1.Node Statistics','By instance','Queries grouped by node instance') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='qtype') FROM rows;
WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('by_city','1.Node Statistics','By city','Queries grouped by city') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='qtype') FROM rows;
WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('by_country','1.Node Statistics','By country','Queries grouped by country') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='qtype') FROM rows;
-- END NODE GROUPING

-- START RSSAC
INSERT INTO dataset (name) VALUES ('load_time');
INSERT INTO dataset (name) VALUES ('zone_size');
WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('load_time','9.RSSAC','Zone propagation time','Latency in the distribution system by serial number') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='load_time') FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('zone_size','9.RSSAC','Zone size','Size of the zone by serial number') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='zone_size') FROM rows;

ALTER table node add ip ipaddress;
-- END RSSAC

-- STRART IP2ANS
WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('asn','7.Client Subnet Statistics','Busiest client ASNs','Client ASNs for the busiest /24(IPv4) or /96(IPv6)') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('bgpprefix','7.Client Subnet Statistics','Busiest client BGP prefix','Clients by BGP prefix for the busiest /24(IPv4) or /96(IPv6)') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('client_addr_vs_rcode_accum_asn','7.Client Subnet Statistics', 'RCODE by client ASNs', 'RCODE by client ASNs for the busiest /24(IPv4) or /96(IPv6)') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET2} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('client_addr_vs_rcode_accum_bgpprefix','7.Client Subnet Statistics', 'RCODE by clients by BGP prefix', 'RCODE by client BGP prefix for the busiest /24(IPv4) or /96(IPv6)')
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET2} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('client_subnet2_accum_asn','8.Classification', 'Query classification by client ASNs', 'Query classification by client ASNs for the busiest /24(IPv4) or /96(IPv6)') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET3} FROM rows;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('client_subnet2_accum_bgpprefix','8.Classification', 'Query classification by clients by BGP prefix', 'Query classification by client BGP prefix for the busiest /24(IPv4) or /96(IPv6)')
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, ${IP2ASN_DATASET3} FROM rows;

UPDATE visible_plot SET title='Busiest client subnets (fixed IPv4/8 or IPv6/32)', display_name='Busiest client subnets (fixed size)' where name='client_subnet_accum';
-- END IP2ASN

-- START DEST ADDR
INSERT INTO dataset (name) VALUES ('server_addr');

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('server_addr','11.Server IP','By server IP address','Server IP address') 
    RETURNING id
) 
INSERT INTO dataset_visible_plot SELECT id, (select id from dataset where name='server_addr') FROM rows;
-- END DEST ADDR

UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
COMMIT;
EOF

echo "`date`: $0 COMPLETED"
#!/bin/bash
#
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000009_ddl_lower_key_index
#

eval $(@LIBEXEC@/conf_read)

DB_NAME=$hhconfig_database_name
DATA_DIR=$hhconfig_directories_data
SQ="'"
PREDICT=0;
SUPPRESS_WARNINGS=0
export PGPASSWORD=$hhconfig_database_owner_pass

set -e
NEW_DDL_SERIAL=12
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=9
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 0 ))

DSC_SCHEMA="dsc"

CONCURRENT=0
SKIP=0

usage () {
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -c Build the indexes with the CONCURRENTLY flag set. This is slower"
    echo "     but allows writes to continue being made to the tables."
    echo "  -s Skip creating the new indexes. The new indexes should improve"
    echo "     performance but are optional. You still need to run this script to"
    echo "     update version numbers correctly in the database."
    echo "  -h Show this help."
    exit 1
}


while getopts ":csh" opt; do
    case $opt in
        c  ) CONCURRENT=1 ;;
        s  ) SKIP=1 ;;
        h  ) usage ;;
        \? ) usage ;;
    esac
done

echo "`date`: $0 COMMENCED"

if [ $SKIP -eq 0 ] ; then
  echo "This script creates new indexes on all the data tables. It will take a long"
  echo "time to complete if you have a lot of data. In addition it requires a lock"
  echo "against writes to the tables being updated (reads are fine) if not run"
  echo "concurrently."
  echo "You should either:"
  echo "1. Stop database writes while this script is running"
  echo "2. Set the -c flag to build the indexes concurrently"
  echo "3. Set the -s flag to skip index creation."
  echo
  echo "See the usage (-h) for more details."
  echo
  echo "Do you wish to continue? [yN]"
  read c
  [[ ! "$c" =~ ^(Y|y) ]] && exit 1
fi

USER=$(whoami)
[ $USER != $hhconfig_database_owner ] && echo "Must be $hhconfig_database_owner to run this script" && exit 1

DB_API_VERSION=`psql -h $hhconfig_database_host $DB_NAME -tc  "select version from dsc.version;"`
[ $DB_API_VERSION != $REQUIRED_API_VERSION ] && echo "Error: Database API version incorrect." && exit 1

LAST_SERIAL=`psql -h $hhconfig_database_host $DB_NAME -tc  "select max(serial) from dsc.internal_version;"`
[ $LAST_SERIAL != $REQUIRED_DDL_SERIAL ] && echo "Error: ddl scripts not run in correct order." && exit 1

# Create node_id indexes on all data tables
if [ $SKIP -eq 0 ] ; then
  MYFILE=$(mktemp)
  if [ $CONCURRENT -eq 0 ] ; then
    psql -h $hhconfig_database_host $DB_NAME -tc "select 'create index idx_' || table_name || '_lower_key1 on ' || table_name || ' (lower(key1));' FROM information_schema.tables where table_name like 'data\_%' and table_type='BASE TABLE' and table_schema='dsc';" > $MYFILE
  else
    psql -h $hhconfig_database_host $DB_NAME -tc "select 'create index CONCURRENTLY idx_' || table_name || '_lower_key1 on ' || table_name || ' (lower(key1));' FROM information_schema.tables where table_name like 'data\_%' and table_type='BASE TABLE' and table_schema='dsc';" > $MYFILE
  fi
  psql -h $hhconfig_database_host $DB_NAME < $MYFILE
  rm $MYFILE
fi

psql -h $hhconfig_database_host $DB_NAME <<EOF
BEGIN;
UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added lower case index to data tables', now() );
COMMIT;
EOF

echo "`date`: $0 COMPLETED"
#!/bin/bash
#
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000013_ddl_dnskey_vs_asn
#

eval $(@LIBEXEC@/conf_read)

DB_NAME=$hhconfig_database_name
DATA_DIR=$hhconfig_directories_data
export PGPASSWORD=$hhconfig_database_owner_pass

NEW_DDL_SERIAL=13
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=10
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))

DSC_SCHEMA="dsc"

usage () {
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
    exit 1
}


while getopts ":h" opt; do
    case $opt in
        h  ) usage ;;
        \? ) usage ;;
    esac
done

# Need a write user to update DB
echo "`date`: $0 COMMENCED"
a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
if [ $? -ne 0 ] ; then
  echo $a
  exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

echo "Starting"
psql -h $hhconfig_database_host $DB_NAME <<EOF
BEGIN;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('dnskey_vs_asn','7.Client Subnet Statistics','DNSKEY Queries by ASN','DNSKEY Queries by ASN') 
    RETURNING id
), rows2 AS (
  INSERT INTO dataset (name) VALUES ('qtype_vs_client_subnet')
    RETURNING id as dataset_id
)
INSERT INTO dataset_visible_plot SELECT id, dataset_id FROM rows, rows2;

UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
COMMIT;
EOF

echo "`date`: $0 COMPLETED"







#!/bin/bash
#
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000014_ddl_service_addr
#


NEW_DDL_SERIAL=14
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=11
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))

DSC_SCHEMA="dsc"

usage () {
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
    exit 1
}


while getopts ":h" opt; do
    case $opt in
        h  ) usage ;;
        \? ) usage ;;
    esac
done

# Need a write user to update DB
echo "`date`: $0 COMMENCED"
a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
if [ $? -ne 0 ] ; then
  echo $a
  exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

echo "Starting"
psql $CONN_STRING <<EOF
BEGIN;
CREATE TABLE $DSC_SCHEMA.service_addr (id SERIAL PRIMARY KEY, server_id INTEGER REFERENCES $DSC_SCHEMA.server (id), address IPADDRESS NOT NULL, UNIQUE (server_id, address));
UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added service address table', now() );
COMMIT;
EOF

echo "`date`: $0 COMPLETED"
#!/bin/bash
#
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000015_ddl_byregion
#


NEW_DDL_SERIAL=15
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=12
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))

DSC_SCHEMA="dsc"

usage () {
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
    exit 1
}


while getopts ":csh" opt; do
    case $opt in
        h  ) usage ;;
        \? ) usage ;;
    esac
done

# Need a write user to update DB
echo "`date`: $0 COMMENCED"
a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
if [ $? -ne 0 ] ; then
  echo $a
  exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

echo "Starting"

psql $CONN_STRING <<EOF
BEGIN;
WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('by_region','1.Node Statistics','By region','Queries by region') 
    RETURNING id
)
INSERT INTO dataset_visible_plot SELECT id, (SELECT id from dataset WHERE name='qtype') FROM rows;

UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new by region graph', now() );
COMMIT;
EOF

echo "`date`: $0 COMPLETED"
#!/bin/bash
#
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000016_ddl_server_addr_vs_trans
#

eval $(@LIBEXEC@/conf_read)

DB_NAME=$hhconfig_database_name
DATA_DIR=$hhconfig_directories_data
export PGPASSWORD=$hhconfig_database_owner_pass

NEW_DDL_SERIAL=16
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=13
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))

DSC_SCHEMA="dsc"

usage () {
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
    exit 1
}


while getopts ":h" opt; do
    case $opt in
        h  ) usage ;;
        \? ) usage ;;
    esac
done

# Need a write user to update DB
echo "`date`: $0 COMMENCED"
a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
if [ $? -ne 0 ] ; then
  echo $a
  exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

echo "Starting"
psql -h $hhconfig_database_host $DB_NAME <<EOF
BEGIN;

WITH rows AS (
  INSERT INTO visible_plot (name, display_category, display_name, title) 
    VALUES ('server_addr_vs_trans','11.Server IP','By server IP address and transport','Server IP address by transport') 
    RETURNING id
), rows2 AS (
  INSERT INTO dataset (name) VALUES ('server_addr_vs_trans')
    RETURNING id as dataset_id
)
INSERT INTO dataset_visible_plot SELECT id, dataset_id FROM rows, rows2;

UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
COMMIT;
EOF

echo "`date`: $0 COMPLETED"







#!/bin/bash
#
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   000017_ddl_node_stats
#


NEW_DDL_SERIAL=17
REQUIRED_DDL_SERIAL=$(( $NEW_DDL_SERIAL - 1 ))
NEW_API_VERSION=14
REQUIRED_API_VERSION=$(( $NEW_API_VERSION - 1 ))

DSC_SCHEMA="dsc"

usage () {
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
    exit 1
}


while getopts ":h" opt; do
    case $opt in
        h  ) usage ;;
        \? ) usage ;;
    esac
done

# Need a write user to update DB
echo "`date`: $0 COMMENCED"
a=$(@LIBEXEC@/check_user_perms -w -C $REQUIRED_API_VERSION -s $REQUIRED_DDL_SERIAL)
if [ $? -ne 0 ] ; then
  echo $a
  exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

echo "Starting"
psql $CONN_STRING <<EOF
BEGIN;
UPDATE visible_plot set display_category='1.Query Statistics' 
  WHERE display_category='1.Node Statistics';
  
UPDATE $DSC_SCHEMA.version SET version = $NEW_API_VERSION;
INSERT INTO $DSC_SCHEMA.internal_version ( serial, script, description, applied ) VALUES ( $NEW_DDL_SERIAL, '$0', 'Added new graphs and datasets for 2.1', now() );
COMMIT;
EOF

echo "`date`: $0 COMPLETED"
This directory contains scripts that are used to update the database
between releases. The database keeps a record of which of these has
been run. They should be run in numerical order.

2.0.0 -> 2.1.0
---------------
To upgrade from 2.0.0 to 2.1.0b1 run the following scripts in this order

* as super user of hedgehog database (e.g. postgres)
  000010_ddl_python

* as the DB write user (e.g. hedgehog):
  000011_ddl_new_graphs
  000012_ddl_lower_key_index
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   check_user_perms
#

# This script should be called like this:
# a=$(@LIBEXEC@/check_user_perms -wfc)
# if [ $? -ne 0 ] ; then
#   echo $a
#   exit 1
# fi
# eval $a
# export PGPASSWORD=$PGPASSWORD


eval $(@LIBEXEC@/conf_read)

usage () {
  echo
  echo "Check if we are a valid read user and check that we can connect to the DB"
  echo "Optionally specify DB version checking, directory permission checking"
  echo "or check for write access to the DB."
  echo
  echo "Usage: $(basename "$0") options"
  echo
  echo "Supported options:"
  echo "  -C <API_VERSION> specify DB API version to check for (implies -c)"
  echo "  -n Indicate that the user doesn't need to access the database"
  echo "  -s <SERIAL_NUMBER> check DB serial number"
  echo "  -c check databse API version against the build time API version"
  echo "  -f check user has write permission to $hhconfig_directories_data"
  echo "  -g check user has write permission to web directory ${hhconfig_directories_rssac%rssac}"
  echo "  -p check user has write permission to plotcache directory $hhconfig_directories_plotcache"
  echo "  -r check user has write permission to rssac directory ${hhconfig_directories_rssac}"
  echo "  -w check we are a write user of the DB"
  echo "  -U specify a DB user"
  echo "  -P specify a DB password"
  echo "  -d run in debug mode"
  echo "  -h Show this help"
}

# Read only access by default
WRITER=0
CHECK_API=0
CHECK_SERIAL=0
CHECK_DATA_DIR=0
CHECK_WEB_DIR=0
CHECK_RSSAC_DIR=0
CHECK_PLOT_CACHE_DIR=0
SPECIFIED_PASS=""
SPECIFIED_USER=""
# By default check for a database connection
CHECK_DB_CONN=1
MY_USER=$(whoami)

REQUIRED_API_VERSION=@DB_VERSION@

while getopts ":C:ncdfgprP:s:U:whv" opt; do
    case $opt in
    C  ) REQUIRED_API_VERSION=$OPTARG
         CHECK_API=1 ;;
    n  ) CHECK_DB_CONN=0 ;;
    c  ) CHECK_API=1 ;;
    d  ) set -x ;;
    f  ) CHECK_DATA_DIR=1 ;;
    g  ) CHECK_WEB_DIR=1 ;;
    p  ) CHECK_PLOT_CACHE_DIR=1 ;;
    P  ) SPECIFIED_PASS=$OPTARG ;;
    p  ) CHECK_RSSAC_DIR=1 ;;
    s  ) CHECK_SERIAL=1
         REQUIRED_DDL_SERIAL=$OPTARG ;;
    U  ) SPECIFIED_USER=$OPTARG ;;
    w  ) WRITER=1 ;;
    h  ) usage
         exit 0 ;;
    v  ) echo "<VERSION>"
         exit 0 ;;
    \? ) echo "Unknown arguments. Use -h for help"
         exit 1 ;;
    esac
done
if [ $CHECK_DB_CONN -eq 1 ] ; then

  USER_NAME=$hhconfig_database_read_user
  if [ -z "$USER_NAME" ]; then
    echo "Database user name not set"
    exit 1
  fi
  export PGPASSWORD="$hhconfig_database_read_pass"
  if [ $WRITER -eq 1 ] ; then
    USER_NAME=$hhconfig_database_owner
    export PGPASSWORD="$hhconfig_database_owner_pass"
  fi
  if [ -n "$SPECIFIED_USER" ] ; then
    USER_NAME=$SPECIFIED_USER
    export PGPASSWORD="$SPECIFIED_PASS"
  fi

  # If the password is set then assume md5 authentication otherwise you must be
  # a user with the same name as the hedgehog database owner and are using peer
  # authentication
  if [ -z "$PGPASSWORD" ] ; then
    if [ "$MY_USER" != "$USER_NAME" ] ; then
      echo "Must be user: '$USER_NAME' and using peer authentication to run this script, currently '$MY_USER'"
      exit 1
    fi
  fi

  PORT=$hhconfig_database_port
  if [ -z "$PORT" ] ; then
    PORT=5432
  fi
  DB_NAME=$hhconfig_database_name
  if [ -z "$DB_NAME" ] ; then
    echo "Database name not set"
    exit 1
  fi
  DB_HOST=$hhconfig_database_host
  if [ -z "$DB_HOST" ]; then
    echo "Database host not set"
    exit 1
  fi

  CONN_STRING="-h $DB_HOST -U $USER_NAME -p $PORT -d $DB_NAME"
  CONN_STRING_XX="host=$DB_HOST port=$PORT dbname=$DB_NAME password=$PGPASSWORD"
  # Test connection to the database
  psql $CONN_STRING -tc "\conninfo" > /dev/null
  if [ $? -ne 0 ] ; then
    echo "Error connecting to database using connection string $CONN_STRING"
    exit 1
  fi

  # Now test the optional parameters...
  TEST=$(psql $CONN_STRING -tc "SELECT has_table_privilege('dsc.version', 'SELECT')" | tr -d '[:space:]')
  if [ $WRITER -eq 1 ] ; then
    TEST=$(psql $CONN_STRING -tc "SELECT has_table_privilege('dsc.version', 'INSERT')" | tr -d '[:space:]')
  fi

  if [ "x$TEST" == "xt" ] ; then
    if [ $CHECK_API -eq 1 ] ; then
      DB_API_VERSION=$(psql $CONN_STRING -tc  "select version from dsc.version;")
      if [ $DB_API_VERSION -ne $REQUIRED_API_VERSION ] ; then
        echo "Error: Database API version incorrect: Found $DB_API_VERSION expected $REQUIRED_API_VERSION"
        exit 1
      fi
    fi
  else
    echo "The user '$USER_NAME' does not have the expected database access."
    exit 1
  fi
fi

if [ $CHECK_SERIAL -eq 1 ] ; then
  LAST_SERIAL=$(psql $CONN_STRING -tc "select max(serial) from dsc.internal_version;")
  if [ $LAST_SERIAL -ne $REQUIRED_DDL_SERIAL ] ; then
    echo "Error: DDL scripts have not been run in correct order. Required $REQUIRED_DDL_SERIAL, actual version is $LAST_SERIAL"
    exit 1
  fi
fi

if [ $CHECK_DATA_DIR -eq 1 ] ; then
  OWNER=$(stat -c %U $hhconfig_directories_data)
  if [ "$OWNER" != "$MY_USER" ] ; then
    echo "The user '$MY_USER'  does not have write access to the data dirctory $hhconfig_directories_data owned by '$OWNER'"
    exit 1
  fi
fi

if [ $CHECK_WEB_DIR -eq 1 ] ; then
  # The %rssac removes rssac from the end of the path, thus checking the www directory instead.
  mydir=${hhconfig_directories_rssac/%rssac/}
  WWW_OWNER=$(stat -c %U $mydir)
  if [ "$WWW_OWNER" != "$MY_USER" ] ; then
    echo "The user '$MY_USER'  does not have write access to the www dirctory $mydir owned by '$WWW_OWNER'"
    exit 1
  fi
fi

if [ $CHECK_PLOT_CACHE_DIR -eq 1 ] ; then
  OWNER=$(stat -c %U $hhconfig_directories_plotcache)
  if [ "$OWNER" != "$MY_USER" ] ; then
    echo "The user '$MY_USER'  does not have write access to the plotcache dirctory $hhconfig_directories_plotcache owned by '$OWNER'"
    exit 1
  fi
fi

if [ $CHECK_RSSAC_DIR -eq 1 ] ; then
  OWNER=$(stat -c %U $hhconfig_directories_rssac)
  if [ "$OWNER" != "$MY_USER" ] ; then
    echo "The user '$MY_USER'  does not have write access to the rssac dirctory $hhconfig_directories_rssac, owned by '$OWNER'"
    exit 1
  fi
fi

# print env variables for eval to process in calling script
echo "RSSAC_DIR=\"$hhconfig_directories_rssac\""
echo "CONF_DIR=\"$hhconfig_directories_conf\""
echo "PLOT_CACHE_DIR=\"$hhconfig_directories_plotcache\""
echo "DATADIR=\"$hhconfig_directories_data\""
echo "PGPASSWORD=\"$PGPASSWORD\""
echo "CONN_STRING=\"$CONN_STRING\""
echo "CONN_STRING_XX=\"$CONN_STRING_XX\""
exit 0
#!/usr/bin/perl -w
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   conf_read
#

use YAML::Tiny;
$yaml = YAML::Tiny->read("@HH_CONF_FILE@");
foreach $key (keys  $yaml->[0]{database}){
  next if (!$yaml->[0]{database}{$key}); 
  print "hhconfig_database_$key=\"$yaml->[0]{database}{$key}\"\n";
}
foreach $key (keys  $yaml->[0]{directories}){
  print "hhconfig_directories_$key=\"$yaml->[0]{directories}{$key}\"\n";
}
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_init
#

eval $(@LIBEXEC@/conf_read)

# DSTART defaults to the current month
DSTART=$(date --date="$(date +%Y-%m-01)" +%Y-%m)
SQ="'"

usage () {
	echo
	echo "Create and populate the database tables with plot options and nodes."
	echo
	echo "This calls the hedeghog_database_update_nodes script which will populate the DB with the"
	echo "specified nodes and create a directory structure for them in the working data directory."
	echo
	echo "Usage: $(basename $0) options"
	echo
	echo "Supported options:"
	echo "  -m Start month of oldest data to be imported (default is current month, format YYYY-MM)"
	echo "  -h Show this help"
}

while getopts ":u:m:ph" opt; do
    case $opt in
        m  ) DSTART=$OPTARG ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

# Validate the input date. Add day since (oddly) date won't parse just year and month
if [[ ! $DSTART =~ ^[0-9]{4}-[0-1][0-9]$ ]] ; then
		echo "Error: The start date is not in the required format of YYYY-MM" 
		exit 1
fi
date "+%Y-%m-%d" --date="${DSTART}-01" >/dev/null 2>&1
is_valid=$?
if [ ${is_valid} -ne 0 ] ; then
	echo "Error: The start date is invalid" 
	exit 1
fi

echo "Running with the following options:"
echo "  Host of database....................... $hhconfig_database_host"
echo "  Port of database....................... $hhconfig_database_port"
echo "  Name of database....................... $hhconfig_database_name"
echo "  Owner of database...................... $hhconfig_database_owner"
echo "  Reader of database..................... $hhconfig_database_read_user"
echo "  Start date of oldest data.............. $DSTART"
echo
  echo  "*** This script will read nodes.csv file in order to  *** "
  echo  "*** populate the database with servers and nodes.     *** "
  echo  "*** Ensure this file is up to date before proceeding. *** "
  echo
  echo -n "Are you sure you want to continue? y/N: "
  read c
  if [ "$c" != "y" ] && [ "$c" != "Y" ]
  then
    echo "Script halted by user - script exiting."
    exit 1
  fi

export PGPASSWORD=$hhconfig_database_owner_pass

psql -h $hhconfig_database_host -d $hhconfig_database_name <<EOF
  REVOKE ALL ON DATABASE "$hhconfig_database_name" FROM public;
  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT SELECT ON TABLES TO hh_read;
  GRANT CONNECT ON DATABASE "$hhconfig_database_name" TO hh_read;
  GRANT USAGE ON SCHEMA dsc to hh_read;
  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT SELECT ON TABLES TO hh_write;
  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT INSERT ON TABLES TO hh_write;
  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT UPDATE ON TABLES TO hh_write;
  ALTER DEFAULT PRIVILEGES FOR ROLE $hhconfig_database_owner IN SCHEMA dsc GRANT DELETE ON TABLES TO hh_write;
  GRANT CONNECT ON DATABASE $hhconfig_database_name TO hh_write;
  GRANT USAGE ON SCHEMA dsc to hh_write;
EOF

#################  CREATE TABLES  #########################
###########################################################
# Create the base tables, views, functions and constraints
echo "Initialising the database (creating base tables, views, functions and constraints)"
psql -h $hhconfig_database_host -d $hhconfig_database_name < @DATA@/hedgehog/sql/initialise_database.sql
echo

# You have to grant privileges on sequences after creating them
echo "Granting privileges on sequences to hh_write"
psql -h $hhconfig_database_host -d $hhconfig_database_name <<EOF
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA dsc TO hh_write;
\q
EOF
echo
###########################################################

###################  POPULATE TABLES  #####################
###########################################################
# Populate the dsc.plot and dsc.iana_lookup tables
echo "Populating lookup tables (dsc.plot, dsc.iana_lookup)"
psql -h $hhconfig_database_host -d $hhconfig_database_name  < @DATA@/hedgehog/sql/insert_lookup_data.sql #> /dev/null
###########################################################

###################  RUN DDL scripts  #####################
###########################################################
# Run any scripts needed to update the DB
echo "Running DDL scripts"
# 000010_ddl not run, as requires super user. Done in DB create.
@LIBEXEC@/DDL_updates/000011_ddl_new_graphs
# DB schema now contains the new index, therefore only run script to update version
@LIBEXEC@/DDL_updates/000012_ddl_lower_key_index -s
# Add a dataset/plot  for dnskey_vs_asn
@LIBEXEC@/DDL_updates/000013_ddl_dnskey_vs_asn
# Add a service address table
@LIBEXEC@/DDL_updates/000014_ddl_service_addr
# Add by region
@LIBEXEC@/DDL_updates/000015_ddl_byregion
# Add server_addr_vs_transport
@LIBEXEC@/DDL_updates/000016_ddl_server_addr_vs_trans
# Remove node from web interface
@LIBEXEC@/DDL_updates/000017_ddl_node_stats
###########################################################

###################  POPULATE TABLES  #####################
###########################################################
echo "Populate servers and nodes"
@LIBEXEC@/database_update_tlds
@LIBEXEC@/database_update_geoip
@LIBEXEC@/database_update_nodes
@LIBEXEC@/database_update_service_addr
###########################################################

#############  PARTITION DATA TABLES  #####################
###########################################################
# Partition the dsc.data table: create 1 table per month per server per plot
echo "Creating partitioned tables for dsc.data (1 table per month per server per plot type)"
echo "This may take some time if there is a lot of historic data to import"
i=1
DNOW_FIRST=`date +%Y%m01`
DNOW=`date +%Y-%m`
DSTART=`date --date="${DSTART}-01" +%Y%m%d`
while [ $DNOW_FIRST -ge $DSTART ]
do
  DNEXT=`date --date="$DNOW_FIRST + 1 month" +%Y-%m`
  @LIBEXEC@/database_manage_partitions -m ${DNOW}
  DNOW_FIRST=`date --date="$DNOW_FIRST - 1 month" +%Y%m%d`
  DNOW=`date --date="$DNOW_FIRST" +%Y-%m`
done
###########################################################
echo "database_init complete"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_manage_partitions

# We need a write user
a=$(@LIBEXEC@/check_user_perms -wc)
if [ $? -ne 0 ] ; then
	echo "$a"
	exit 1
fi
eval "$a"
export PGPASSWORD=$PGPASSWORD

usage () {
	echo
	echo "Create a new set of partitioned data tables in the database to hold imported data."
	echo "The partitions are created per month of data."
	echo
	echo "Usage: $(basename "$0") options"
	echo
	echo "Supported options:"
	echo " -m Month to create partitions for (default is next month, format YYYY-MM)."
	echo " -h Show this help."
}

while getopts ":m:h" opt; do
    case $opt in
        m  ) PDATE=$OPTARG ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

if [ -z "$PDATE" ] ; then
    NOW=$(date --date="+1 month" +%Y_%m)
    NEXT=$(date --date="+2 month" +%Y_%m)
else
	if [[ ! $PDATE =~ ^[0-9]{4}-[0-1][0-9]$ ]] ; then
			echo "Error: The start date is not in the required format of YYYY-MM" 
			exit 1
	fi
	# Validate the input date. Add day since (oddly) date won't parse just year and month
	date "+%Y-%m-%d" --date="${PDATE}-01" >/dev/null 2>&1
	is_valid=$?
	if [ ${is_valid} -ne 0 ] ; then
		echo "Error: The start date is invalid" 
		exit 1
	fi
	
    NOW=$(date --date="$PDATE-01" +%Y_%m)
    
    IFS='_' read -a YM <<< "$NOW"
    YEAR=${YM[0]}
    MONTH=${YM[1]}
    NEXT="${YEAR}_$(( ${MONTH#0} + 1))"

    if [ ${MONTH#0} -eq 12 ] ; then
        NEXT=$((YEAR + 1))"_01"
    fi
fi

SERVERS=$(psql $CONN_STRING -tc "SELECT name FROM dsc.server;")
PLOTS=$(psql $CONN_STRING -tc "SELECT name FROM dsc.dataset;")

for server in $SERVERS
do
    s_id=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server WHERE name = '$server';")
    for plot in $PLOTS
    do
      p_id=$(psql $CONN_STRING -tc "SELECT id FROM dsc.dataset WHERE name = '$plot';")
      
      psql $CONN_STRING <<EOF
BEGIN;

CREATE TABLE dsc.data_${server}_${plot}_${NOW} (
  CHECK (starttime >= TIMESTAMP '$NOW-01 00:00:00' AT TIME ZONE 'UTC' AND starttime < TIMESTAMP '$NEXT-01 00:00:00' AT TIME ZONE 'UTC'),
  CHECK (plot_id = $p_id),
  CHECK (server_id = $s_id)
) INHERITS (dsc.data);

ALTER TABLE dsc.data_${server}_${plot}_${NOW}
        ADD CONSTRAINT pk_data_${server}_${plot}_${NOW} PRIMARY KEY (server_id, node_id , plot_id , starttime , key1, key2 ),
        ADD CONSTRAINT fk_data_${server}_${plot}_${NOW}_node FOREIGN KEY (node_id) REFERENCES dsc.node (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION,
        ADD CONSTRAINT fk_data_${server}_${plot}_${NOW}_plot FOREIGN KEY (plot_id) REFERENCES dsc.dataset (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION;
CREATE INDEX idx_data_${server}_${plot}_${NOW}_node_id ON dsc.data_${server}_${plot}_${NOW} (node_id);
COMMIT;
EOF
    done
done

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_process_rssac_data
#

# We need a write user
a=$(@LIBEXEC@/check_user_perms -wc)
if [ $? -ne 0 ] ; then
	echo "$a"
	exit 1
fi
eval "$a"
export PGPASSWORD=$PGPASSWORD

DEFAULT_DAY=$(date --date="- 7 day" +%Y-%m-%d)
START=${DEFAULT_DAY}
FORCE_SUM_DEL=0
DELETE_RAW=0

usage () {
    echo
    echo "Process the raw RSSAC unique source data into a summary table for a specified 24 hour period"
    echo "(Must be run prior to generating the RSSAC reports for this day with the rssac_generate_reports script)"
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -d Date on which to process data (default is 7 days ago: \"${DEFAULT_DAY}\", format: YYYY-MM-DD)"
    echo "  -f Force overwrite of existing summary data"
    echo "  -D Also delete the raw unique_source data from the database to save space (default: no delete)"
    echo "  -h Show this help"
}

while getopts ":d:fDh" opt; do
    case $opt in
        d  ) START=$OPTARG ;;
        f  ) FORCE_SUM_DEL=1 ;;
        D  ) DELETE_RAW=1 ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"


# Validate the start date
if [[ ! $START =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]$ ]] ; then
		echo "Error: The start date is not in the required format of YYYY-MM-DD" 
		exit 1
fi
date "+%Y-%m-%d" --date="${START}" >/dev/null 2>&1
is_valid=$?
if [ ${is_valid} -ne 0 ] ; then
	echo "Error: The start date is invalid" 
	exit 1
fi

echo "Processing unique_source data for ${START}"

# Confirm force flag as this should be exceptional
if [ ${FORCE_SUM_DEL} -eq 1 ] ; then
	 echo
	 echo "You have specified the -f flag to force an overwrite of existing summary data"
	 echo -n "Are you sure you want to continue? y/N: "
	 read c
	 if [ "$c" != "y" ] && [ "$c" != "Y" ]
	 then
	   echo "Script halted by user - script exiting."
	   exit 1
	 fi
fi

SQ="'"
SERVER_IDS=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server;")
UNIQUE_SOURCE_RAW_PLOT_NAME="unique_sources_raw"
UNIQUE_SOURCE_SUM_PLOT_NAME="unique_sources"
UNIQUE_SOURCE_RAW_PLOT_ID=$(psql $CONN_STRING -tc "SELECT plot_id FROM dsc.plot where name=${SQ}${UNIQUE_SOURCE_RAW_PLOT_NAME}${SQ};")
UNIQUE_SOURCE_SUM_PLOT_ID=$(psql $CONN_STRING -tc "SELECT plot_id FROM dsc.plot where name=${SQ}${UNIQUE_SOURCE_SUM_PLOT_NAME}${SQ};")

DELETE_SUMMARY_SQL=""
DELETE_RAW_SQL=""
IPV4_key="IPv4"
IPV6_key="IPv6"
IPV6_ag_key="IPv6/64"
IPV4_value=0
IPV6_value=0
IPV6_ag_value=0

for SERVER_ID in $SERVER_IDS ;do

	# Set up the table name
    SERVER_NAME=$(psql $CONN_STRING -tc "SELECT name FROM dsc.server WHERE id=${SERVER_ID};" | tr -d [[:space:]])
    NODE_COUNT=$(psql $CONN_STRING -tc "SELECT count(*) FROM dsc.node where server_id=${SERVER_ID};")
    NODE_IDS=$(psql $CONN_STRING -tc "SELECT id FROM dsc.node where server_id=${SERVER_ID} order by id;")
    MONTH=$(date --date="${START}" +%m)
    YEAR=$(date --date="${START}" +%Y)
    # Could do this work directly on dsc.data instead
    UNIQUE_SOURCE_RAW_TABLE_NAME="dsc.data_${SERVER_NAME}_${UNIQUE_SOURCE_RAW_PLOT_NAME}_${YEAR}_${MONTH}"
    UNIQUE_SOURCE_SUM_TABLE_NAME="dsc.data_${SERVER_NAME}_${UNIQUE_SOURCE_SUM_PLOT_NAME}_${YEAR}_${MONTH}"
    echo "Found $NODE_COUNT nodes for server $SERVER_NAME" 


   for NODE_ID in $NODE_IDS ; do
	
		echo "Processing node id $NODE_ID"
		# First up, check that there isn't an entry in the summary table for this node. 
		# Note that if there is actually data when the transaction occurs then the insert will fail and the transaction will rollback.
		SUMMARY_TABLE_ENTRY=$(psql $CONN_STRING -tc "SELECT count(*) FROM ${UNIQUE_SOURCE_SUM_TABLE_NAME} WHERE node_id=${NODE_ID} and starttime = TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC' ;")
		if [ ${SUMMARY_TABLE_ENTRY} -ne 0 ] ; then
			if [ ${FORCE_SUM_DEL} -eq 1 ] ; then
				echo "Summary data already exists for node id = ${NODE_ID}... Force flag was used overwriting summary data!"
				DELETE_SUMMARY_SQL="DELETE FROM ${UNIQUE_SOURCE_SUM_TABLE_NAME} WHERE node_id=${NODE_ID} and starttime = TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC' ;"
			else
				echo "Summary data already exists for node id = ${NODE_ID}... skipping"
				continue;
			fi
		fi
		
		# Now lets get the summary numbers from the raw table if they exist. Reset the counters to 0 first.
		IPV4_value=0
		IPV6_value=0
		IPV6_ag_value=0
		IPv4_SUMMARY_RESULT=$(psql $CONN_STRING -tc "SELECT y from unique_source_summary_function(${SERVER_ID}, ${UNIQUE_SOURCE_RAW_PLOT_ID}, TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', TIMESTAMP '${START} 23:59:00' AT TIME ZONE 'UTC', '${NODE_ID}') WHERE x='${IPV4_key}';")
		IPv6_SUMMARY_RESULT=$(psql $CONN_STRING -tc "SELECT y from unique_source_summary_function(${SERVER_ID}, ${UNIQUE_SOURCE_RAW_PLOT_ID}, TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', TIMESTAMP '${START} 23:59:00' AT TIME ZONE 'UTC', '${NODE_ID}') WHERE x='${IPV6_key}';")
		IPv6_ag_SUMMARY_RESULT=$(psql $CONN_STRING -tc "SELECT y from unique_source_summary_function(${SERVER_ID}, ${UNIQUE_SOURCE_RAW_PLOT_ID}, TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', TIMESTAMP '${START} 23:59:00' AT TIME ZONE 'UTC', '${NODE_ID}') WHERE x='${IPV6_ag_key}';")
		if [ ! -z "${IPv4_SUMMARY_RESULT}" ] ; then IPV4_value=${IPv4_SUMMARY_RESULT} ; fi
		if [ ! -z "${IPv6_SUMMARY_RESULT}" ] ; then IPV6_value=${IPv6_SUMMARY_RESULT} ; fi
		if [ ! -z "${IPv6_ag_SUMMARY_RESULT}" ] ; then IPV6_ag_value=${IPv6_ag_SUMMARY_RESULT} ; fi
								
		# Time to delete the raw data if that is what we were told to do						
		if [ ${DELETE_RAW} -eq 1 ] ; then
			echo "Deleting raw data for for node id = ${NODE_ID}"
			DELETE_RAW_SQL="DELETE FROM ${UNIQUE_SOURCE_RAW_TABLE_NAME} WHERE node_id=${NODE_ID} and starttime = TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC' ;"
		fi
													
		psql $CONN_STRING <<EOF
BEGIN;
set client_min_messages to WARNING;
${DELETE_SUMMARY_SQL}
INSERT INTO ${UNIQUE_SOURCE_SUM_TABLE_NAME} (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', ${SERVER_ID}, ${NODE_ID}, ${UNIQUE_SOURCE_SUM_PLOT_ID}, ${SQ}${IPV4_key}${SQ}, 1, ${IPV4_value});
INSERT INTO ${UNIQUE_SOURCE_SUM_TABLE_NAME} (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', ${SERVER_ID}, ${NODE_ID}, ${UNIQUE_SOURCE_SUM_PLOT_ID}, ${SQ}${IPV6_key}${SQ}, 1, ${IPV6_value});
INSERT INTO ${UNIQUE_SOURCE_SUM_TABLE_NAME} (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES (TIMESTAMP '${START} 00:00:00' AT TIME ZONE 'UTC', ${SERVER_ID}, ${NODE_ID}, ${UNIQUE_SOURCE_SUM_PLOT_ID}, ${SQ}${IPV6_ag_key}${SQ}, 1, ${IPV6_ag_value});
${DELETE_RAW_SQL}
COMMIT;
EOF
	done
done
echo "$(date): $0 COMPLETED"
exit 0
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_rm_old_rssac_data
#

#TODO: Add option to select all servers or just one server

# We need a write user
a=$(@LIBEXEC@/check_user_perms -wc)
if [ $? -ne 0 ] ; then
	echo "$a"
	exit 1
fi
eval "$a"
export PGPASSWORD=$PGPASSWORD

DEFAULT_DAY=$(date --date="$(date +%Y-%m-15)" +%Y%m%d)
NOW=$(date --date="now" +%Y%m%d)

if [ $DEFAULT_DAY -ge $NOW ] ; then
	START=$(date --date="$NOW - 2 month" +%Y-%m)
else
	START=$(date --date="$NOW - 1 month" +%Y-%m)
fi


usage () {
    echo
    echo "Delete all the raw RSSAC unique_source data in the database for a given month. All data older"
    echo "than this is also deleted."
    echo "This script is intended to be run monthly to free disk space on systems where the raw"
    echo "RSSAC unique sources data becomes unmanagably large."
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -m Month from which to delete all raw unique source data (default ${START}: format YYYY-MM)"
    echo "       (default is 2 months ago when in the first half of the current month and"
    echo "        1 month ago when in the second half of the current month)"
    echo "  -h Show this help"
}

while getopts ":m:h" opt; do
    case $opt in
        m  ) START=$OPTARG ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

# Validate the input date. Add day since (oddly) date won't parse just year and month
if [ ! -z "$START" ] ; then
	if [[ ! $START =~ ^[0-9]{4}-[0-1][0-9]$ ]] ; then
			echo "Error: The start date is not in the required format of YYYY-MM" 
			exit 1
	fi
	date "+%Y-%m-%d" --date="${START}-01" >/dev/null 2>&1
	is_valid=$?
	if [ ${is_valid} -ne 0 ] ; then
		echo "Error: The start date is not valid" 
		exit 1
	fi
	
	START_FIRST=$(date --date=$START-01 +%Y%m%d)
	NOW_FIRST=$(date --date="$NOW" +%Y%m01)
	if [ $NOW_FIRST -le $START_FIRST ] ; then
		echo "Error: The given month cannot be this month or in the future." 
		exit 1
	fi
fi

echo "Deleting unique_source data during and before ${START}" 

UNIQUE_SOURCE_RAW_PLOT_NAME="unique_sources_raw"

TABLE_NAMES=$(psql $CONN_STRING -tc "select tablename from pg_tables where schemaname='dsc' order by tablename" | grep $UNIQUE_SOURCE_RAW_PLOT_NAME)

START_TRUNC=$(date --date="$START-01" +%Y%m%d)
for TABLENAME in $TABLE_NAMES
do
	(( SIZE=${#TABLENAME}-7 ))
	TMP_DATE=$(echo ${TABLENAME:$SIZE} | sed -e 's/_/-/')
	TABLE_DATE=$(date --date="$TMP_DATE-01" +%Y%m%d)
	if [ $START_TRUNC -ge $TABLE_DATE ] ; then
		echo "Truncating table $TABLENAME"
		psql $CONN_STRING -c "truncate $TABLENAME;"
	fi
done

echo "$(date): $0 COMPLETED"
exit 0
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_update_geoip
#

# We need a write user
a=$(@LIBEXEC@/check_user_perms -wc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

TMP_DIR=/tmp

usage () {
	echo
	echo "Update the GeopIP list in the database. This reads input from ${TMP_DIR}"
	echo 
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
}

while getopts ":h" opt; do
    case $opt in
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

cd $TMP_DIR
rm -f GeoLite2-City-CSV.zip.md5
curl -O http://geolite.maxmind.com/download/geoip/database/GeoLite2-City-CSV.zip.md5
if [ -f GeoLite2-City-CSV.zip ] ; then
  md5sum GeoLite2-City-CSV.zip | awk ' { print $1 } ' | diff -w GeoLite2-City-CSV.zip.md5 -
  if [ $? -ne 0 ] ; then
    rm GeoLite2-City-CSV.zip
  fi
fi
if [ ! -f GeoLite2-City-CSV.zip ] ; then
  curl -O http://geolite.maxmind.com/download/geoip/database/GeoLite2-City-CSV.zip
  md5sum GeoLite2-City-CSV.zip | awk ' { print $1 } ' | diff -w GeoLite2-City-CSV.zip.md5 -
  if [ $? -ne 0 ] ; then
    echo "Error downloading and md5-ing GeoLite2-City-CSV.zip"
    exit 1
  fi
fi
  
unzip -oj GeoLite2-City-CSV.zip

echo "Preparing City Blocks data..."
@LIBEXEC@/utility_prepare_GeoLite2-City-Blocks-IPvX

echo "Preparing Location data..."
@LIBEXEC@/utility_prepare_GeoLite2-City-Locations-en
			
psql $CONN_STRING << EOF                                                                                                                                          
BEGIN;
delete from geoip;
\copy geoip (ip_range, geoname, latitude, longitude) from '/tmp/blocks_ready_to_import.csv' (HEADER false, DELIMITER ',', FORMAT 'csv')
delete from locations;
\copy locations (geoname_id, name, city) from '/tmp/locations_ready_to_import.csv' (HEADER false, DELIMITER ',', FORMAT 'csv')
COMMIT;
EOF

rm -f /tmp/blocks_ready_to_import.csv /tmp/locations_ready_to_import.csv

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_update_nodes
#

eval $(@LIBEXEC@/conf_read)
NODES_FILE="${hhconfig_directories_conf}/nodes.csv"
DATA_DIR=$hhconfig_directories_data
if [ ! -e $NODES_FILE ] ; then
	echo "Error attempting to find nodes.csv. Tried: $NODES_FILE"
	exit 1
fi

# We need a write user with access to the data dir. 
a=$(@LIBEXEC@/check_user_perms -wfc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

usage () {
    echo
    echo "Add/update the servers and nodes in the database. This reads input from file"
    echo "called nodes.csv in the $hhconfig_directories_conf directory."
    echo " - An example nodes.csv file is installed if one does not exist."
    echo " - See the comments in that file for details of the format."
    echo " - No action is taken on servers/nodes that are in the database but are not in the input files."
    echo 
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -r rename/remove flag     This enables processing of the rename_or_remove field in nodes.csv"
    echo "                            If this flag is not given any lines in the input file that have"
    echo "                            text in the rename_or_remove field are ignored."
    echo "                            Renaming a node will update the database and move the data "
    echo "                            directory to the new name."
    echo "                            Removing a node will delete all associated data in the database"
    echo "                            but leave any data on disk untouched (for manual removal)."
    echo "                            Removing nodes could take a long time if you have a lot of data."
    echo "  -d debug mode (set -x)."
    echo "  -h Show this help."
}
RENAME=0
while getopts ":rhd" opt; do
    case $opt in
        r  ) RENAME=1 ;;
        h  ) usage
             exit 1 ;;
        d  ) set -x ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

# Read CSV file with the following fields in order:
fields=(server_name node_name node_region node_country node_city node_subgroup node_ip old_name)

NAME_REGEX="^[a-zA-Z0-9\\.-]+$"
NAME_SPACE_REGEX="^[[:space:]a-zA-Z0-9\\-]+$"
NAME_ERROR="It may contain alphanumeric characters, hyphens (-) and full stops (.) only."
NAME_SPACE_ERROR="It may contain alphanumeric characters, spaces and hyphens (-) only."
IP_REGEXP="^[a-fA-F0-9:\\.]*$"
IP_ERROR="It must be a valid IPv4 or IPv6 address."
# This restriction comes from the maximum table name length.
MAX_SERVERNAME_LEN=11


declare -A aNODES
declare -A aFIELD_OPTIONAL
declare -A aFIELD_REGEX
declare -A aFIELD_ERROR

aFIELD_OPTIONAL["server_name"]="no"
aFIELD_OPTIONAL["node_name"]="no"
aFIELD_OPTIONAL["node_region"]="yes"
aFIELD_OPTIONAL["node_country"]="yes"
aFIELD_OPTIONAL["node_city"]="yes"
aFIELD_OPTIONAL["node_subgroup"]="yes"
aFIELD_OPTIONAL["node_ip"]="yes"
aFIELD_OPTIONAL["old_name"]="yes"

aFIELD_REGEX["server_name"]=$NAME_REGEX
aFIELD_REGEX["node_name"]=$NAME_REGEX
aFIELD_REGEX["node_region"]=$NAME_REGEX
aFIELD_REGEX["node_country"]=$NAME_SPACE_REGEX
aFIELD_REGEX["node_city"]=$NAME_SPACE_REGEX
aFIELD_REGEX["node_subgroup"]=$NAME_REGEX
aFIELD_REGEX["node_ip"]=$IP_REGEXP
aFIELD_REGEX["old_name"]=$NAME_REGEX

aFIELD_ERROR["server_name"]=$NAME_ERROR
aFIELD_ERROR["node_name"]=$NAME_ERROR
aFIELD_ERROR["node_region"]=$NAME_ERROR
aFIELD_ERROR["node_country"]=$NAME_SPACE_ERROR
aFIELD_ERROR["node_city"]=$NAME_SPACE_ERROR
aFIELD_ERROR["node_subgroup"]=$NAME_ERROR
aFIELD_ERROR["node_ip"]=$IP_ERROR
aFIELD_ERROR["old_name"]=$NAME_ERROR

i=0
NEW_SERVER=0
NODES_UPDATED=0
NODES_ADDED=0
NODES_ERRORS=0
NODES_RENAMED=0
NODES_REMOVED=0
echo
echo "PROCESSING:   SERVER, NODE, GROUP, COUNTRY, CITY, INSTANCE, IP ADDRESS, OLD NAME"
# Read nodes.csv
while read line || [[ -n "$line" ]]; do
	i=$(($i+1))
	trimmed_line=$(echo $line | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')

	if [[ -z ${trimmed_line} ]] || [[ "${trimmed_line}" =~ ^# ]] ; then
		continue
	fi	 

	IFS=, read -r aNODES["server_name"] aNODES["node_name"] \
	                aNODES["node_region"] aNODES["node_country"] \
	                aNODES["node_city"] aNODES["node_subgroup"] \
	                aNODES["node_ip"] aNODES["old_name"] \
	                junk <<< "$trimmed_line"
	
	echo
	if [[ -n "$junk" ]] ; then
		echo "***ERROR***:  Line $i, Too many fields. This line will be ignored."
		((NODES_ERRORS++))
		continue
	fi

	error="no"
	# Turn off filename expansion in case someone has put something stupid 
	# in the nodes file
	set -f
	for key in ${fields[*]} ; do
		aNODES["$key"]=$(echo ${aNODES["$key"]} | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
		if [[ ${aNODES["$key"]} =~ ^[[:space:]]*$ ]] ; then
			# Empty IP address MUST be null not an empty string
			if [ "$key" == "node_ip" ] ; then
				aNODES["$key"]="null"
			fi
			if [ "${aFIELD_OPTIONAL["$key"]}" == "no" ] ; then
				echo "***ERROR***:  Line $i, Field $key is not optional."
				((NODES_ERRORS++))
				error="yes"
				break
			else
				continue
			fi
		fi
		if [ $RENAME -eq 0 ] && [ -n "${aNODES["old_name"]}" ] ; then
			echo "***ERROR***:  Line $i, old-name field is not empty: contains \"${aNODES["old_name"]}\" but -r (rename) flag is not set."
			((NODES_ERRORS++))
			error="yes"
			break
		fi
		if [[ ! ${aNODES["$key"]} =~ ${aFIELD_REGEX["$key"]} ]] ; then
			echo "***ERROR***:  Line $i, Field $key value: \"${aNODES["$key"]}\" is not valid. ${aFIELD_ERROR["$key"]} This line will be ignored."
			((NODES_ERRORS++))
			error="yes"
			break
		fi
	done
	if [ "$error" == "yes" ] ; then
		continue
	fi
	set +f
	
	echo -n "PROCESSING:   ${aNODES["server_name"]}, ${aNODES["node_name"]}, ${aNODES["node_region"]}, ${aNODES["node_country"]}, ${aNODES["node_city"]}, ${aNODES["node_subgroup"]}"
	if [ ${aNODES["node_ip"]} != "null" ] ; then
		echo -n ", ${aNODES["node_ip"]}"
	else
		echo -n ","
	fi
	if [ -n "${aNODES["old_name"]}" ] ; then
		echo ", ${aNODES["old_name"]}"
	else
		echo ","
	fi
	
	SERVER_ID=$(psql $CONN_STRING -tc "select id from server where display_name='${aNODES["server_name"]}'")
	if [ -z "$SERVER_ID" ]; then
		# lower case
		SERVER_DB_NAME=${aNODES["server_name"]}
		SERVER_DB_NAME=${SERVER_DB_NAME,,}
		# subsitute _ for -
		SERVER_DB_NAME=${SERVER_DB_NAME//-/_}
		# subsitute __ for .
		SERVER_DB_NAME=${SERVER_DB_NAME//./__}
		if [ ${#SERVER_DB_NAME} -gt $MAX_SERVERNAME_LEN ] ; then
			echo "***ERROR***:  Line $i, The Server name ${aNODES["server_name"]} is too long to be stored internally in Hedgehog."
			echo "              Names containing only alphanumeric characters and hyphens can be a maximum of $MAX_SERVERNAME_LEN characters long."
			echo "              Names also containing full stops must be less than this. This line will be ignored."
			((NODES_ERRORS++))
			continue
		fi
		SERVER_ID=$(psql $CONN_STRING -tqc "INSERT INTO dsc.server (name, display_name) VALUES ('$SERVER_DB_NAME', '${aNODES["server_name"]}') RETURNING id;")
		echo "ADDED SERVER: ${aNODES["server_name"]}"
		((NEW_SERVER++))
	fi
	
	if [ $RENAME -eq 1 ] && [ -n "${aNODES["old_name"]}" ] ; then
		NODE_ID=$(psql $CONN_STRING -tc "select id from dsc.node where server_id=$SERVER_ID and name='${aNODES["old_name"]}'")
		if [ "${aNODES["old_name"]}" == "remove" ] ; then
			NODE_ID=$(psql $CONN_STRING -tc "select id from dsc.node where server_id=$SERVER_ID and name='${aNODES["node_name"]}'")
			if [ -n "$NODE_ID" ] ; then
				psql $CONN_STRING -c "BEGIN; DELETE FROM dsc.data where node_id = ${NODE_ID}; DELETE FROM node where id = ${NODE_ID}; COMMIT;"
				((NODES_REMOVED++))
				echo "The data directory for Server:${aNODES["server_name"]} Node: ${aNODES["node_name"]} can now be manually removed."
				continue
			else
				echo "***ERROR***:  Line $i, The node called ${aNODES["node_name"]} could not be found"
				((NODES_ERRORS++))
				continue
			fi
		elif [ -z "$NODE_ID" ] ; then
			echo "***ERROR***:  Line $i, The node called ${aNODES["old_name"]} could not be found"
			((NODES_ERRORS++))
			continue
		else
			psql $CONN_STRING -c "UPDATE dsc.node SET name='${aNODES["node_name"]}' WHERE id=${NODE_ID};"
			((NODES_RENAMED++))
		fi
	else
		NODE_ID=$(psql $CONN_STRING -tc "select id from dsc.node where server_id=$SERVER_ID and name='${aNODES["node_name"]}'")
	fi
	if [ ${aNODES["node_ip"]} != "null" ] ; then
		aNODES["node_ip"]="'${aNODES["node_ip"]}'"
	fi
	if [ -z "$NODE_ID" ] ; then
		NODE_ID=$(psql $CONN_STRING -tqc "INSERT INTO dsc.node (server_id, name, region, country, city, subgroup, ip) VALUES ($SERVER_ID, '${aNODES["node_name"]}', '${aNODES["node_region"]}', '${aNODES["node_country"]}', '${aNODES["node_city"]}', '${aNODES["node_subgroup"]}', ${aNODES["node_ip"]});")
		echo "ADDED NODE:   ${aNODES["server_name"]}, ${aNODES["node_name"]}"
		((NODES_ADDED++))
	else
		# It would be nice here to only update when something has changed so the user can easily see what changed
		psql $CONN_STRING -c "UPDATE dsc.node SET server_id=$SERVER_ID, name='${aNODES["node_name"]}', region='${aNODES["node_region"]}', country='${aNODES["node_country"]}', city='${aNODES["node_city"]}', subgroup='${aNODES["node_subgroup"]}', ip=${aNODES["node_ip"]} where id=$NODE_ID;"
		echo "UPDATED NODE: ${aNODES["server_name"]}, ${aNODES["node_name"]}"
		((NODES_UPDATED++))
	fi
	if [ $RENAME -eq 1 ] && [ -n "${aNODES["old_name"]}" ] && [ "${aNODES["old_name"]}" != "remove" ] ; then
		if [ ! -d $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]} ] ; then
			echo "Error: Unable to find $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]}."
			((NODES_ERRORS++))
			error="yes"
			break
		elif [ -d $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]} ] ; then
			echo "Error: New directory $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]} already exists."
			((NODES_ERRORS++))
			error="yes"
			break
		else
			mv -vT $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]} $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]}
			if [ $? -ne 0 ] ; then
				echo "Error: Failed to move $DATA_DIR/${aNODES["server_name"]}/${aNODES["old_name"]}."
				((NODES_ERRORS++))
				error="yes"
				break
			fi
		fi
	else
		mkdir -p $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]}/incoming
		chmod 775 $DATA_DIR/${aNODES["server_name"]}/${aNODES["node_name"]}/incoming
	fi
	junk=""
done < ${NODES_FILE}

	echo 
	echo "SUMMARY:"
	echo " - Servers added    : $NEW_SERVER"
	echo " - Nodes added      : $NODES_ADDED"
	echo " - Nodes updated    : $NODES_UPDATED"
	echo " - Nodes renamed    : $NODES_RENAMED"
	echo " - Nodes removed    : $NODES_REMOVED"
	echo " - Nodes with errors: $NODES_ERRORS"
	echo

if [ ${NEW_SERVER} -ge 1 ] ; then
	echo "********************************************************************************"
	echo "New servers have been added to Hedgehog! To enable the import of data for this"
	echo "server the 'manage_partitions' script MUST be run manually now."
	echo "********************************************************************************"
	echo
fi

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_update_service_addr
#

eval $(@LIBEXEC@/conf_read)
SERVICE_ADDR_FILE="${hhconfig_directories_conf}/service_addr.csv"
if [ ! -e $SERVICE_ADDR_FILE ] ; then
	echo "Error attempting to find service_addr.csv. Tried: $SERVICE_ADDR_FILE"
	exit 1
fi

# We need a write user
a=$(@LIBEXEC@/check_user_perms -wc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

DELETE=0

usage () {
    echo
    echo "Update the Server service addreses list in the database."
    echo "This list is only used to filter which addresses are displayed in the"
    echo "'By server IP address' plot. If there are any addresses in the list"
    echo "then only these addresses are displayed. If there are no entries "
    echo "in the list then all the addresses for which there is data are displayed."
    echo "By default this script reads the $SERVICE_ADDR_FILE file and adds"
    echo "the addresses found there."
    echo "After running this script it is advisable to remove any cached plots"
    echo "for the 'By server IP address' plot by using the plotcache_rm_cached_plots"
    echo "script."
    echo 
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -d Delete all existing entries in the server service address list"
    echo "  -h Show this help."
}

while getopts ":dh" opt; do
    case $opt in
        d  ) DELETE=1 ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"
if [ ${DELETE} -eq 1 ] ; then
    psql $CONN_STRING << EOF
DELETE FROM service_addr;
EOF
echo "$(date): $0 COMPLETED"
exit 0
fi

i=0
while read line || [[ -n "$line" ]]; do
    i=$(($i+1))
    trimmed_line=$(echo $line | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')

    if [[ -z ${trimmed_line} ]] || [[ "${trimmed_line}" =~ ^# ]] ; then
            continue
    fi
    IFS=, read -r SERVER_NAME ADDRESS junk <<< "$trimmed_line"
    if [[ -n $junk ]] ; then
            echo "***ERROR***:  Line $i, Too many fields. This line will be ignored."
            continue
    fi
    echo $SERVER_NAME -- $ADDRESS
psql $CONN_STRING << EOF
BEGIN;
insert into service_addr (server_id, address) select id, '$ADDRESS' from server where display_name = '$SERVER_NAME';
COMMIT;
EOF
    junk=""
done < ${SERVICE_ADDR_FILE}

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_update_tlds
#

# We need a write user
a=$(@LIBEXEC@/check_user_perms -wc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

TLDS_DIR=$CONF_DIR/tlds.d
TLD_TMP="/tmp/hedgehog_tld_inserts.sql"
COUNT=0

usage () {
	echo
	echo "Update the TLD list in the database. This reads input from files"
	echo "from the $TLDS_DIR directory."
	echo "No action is taken on TLDs that are in the database but are not in the input files."
	echo 
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
}

while getopts ":h" opt; do
    case $opt in
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

if [ ! -d "$TLDS_DIR" ] ; then
	echo "No tlds.d dir found. Expecting a dir $TLDS_DIR... done..."
	exit 1
fi

echo "insert into tlds (alabel, type)" > $TLD_TMP

for TLD_FILE in ${TLDS_DIR}/* ; do
  # Process the tld file
  echo "Processing file: " $TLD_FILE
  while read line || [[ -n "$line" ]]; do
  	trimmed_line=$(echo $line | sed -e 's/^ *//' -e 's/ *$//')

  	if [[ -z ${trimmed_line} ]] || [[ "${trimmed_line}" =~ ^# ]] ; then
  		continue
  	fi	
	
  	OIFS=$IFS
  	IFS=','
  	read -ra arrline <<< "$trimmed_line"
  	IFS=$OIFS
  	ALABEL=$(echo ${arrline[0]} | sed -e 's/^ *//' -e 's/ *$//')
    TLDTYPE=$(echo ${arrline[1]} | sed -e 's/^ *//' -e 's/ *$//')
    
  	# First, validate the input	
  	if [ -z "$ALABEL" ] ; then
  		echo "Missing alabel. This entry will be ignored."
  		continue
  	fi	
	
  	if ! [[ "${ALABEL}" =~ ^[a-zA-Z0-9\-]+$ ]] ; then
  		echo "alabel name '${ALABEL}' is invalid. It may contain alphanumeric characters and hyphens (-) only. This entry will be ignored."
  		continue
  	fi
    
  	if [ -z "$TLDTYPE" ] ; then
    	echo "Missing TLD type field for alabel $ALABEL - alabel ignored"
    	continue
    fi
    
    if [ $COUNT -gt 0 ]; then
      echo "UNION" >> $TLD_TMP
    fi
    echo "select lower('${ALABEL}'), id from tld_types where type_name='${TLDTYPE}'" >> $TLD_TMP
    (( COUNT+=1 ))
  done < ${TLD_FILE}
done
echo ";" >> $TLD_TMP
psql $CONN_STRING -tf $TLD_TMP

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_update_tlds_from_zone
#

# We need a write user
a=$(@LIBEXEC@/check_user_perms -wc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

TLD_TMP="/tmp/hedgehog_tld_inserts_from_zone.sql"

usage () {
	echo
	echo "Update the TLD list in the database. This reads input from IANA ftp site"
	echo "from the $TLDS_DIR directory."
	echo "No action is taken on TLDs that are in the database but are not in the input data."
	echo 
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -h Show this help."
}

while getopts ":h" opt; do
    case $opt in
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"
rm $TLD_TMP

TTYPE=$(psql $CONN_STRING -tc "select id from tld_types where type_name='New-gTLD'")
TCOUNT1=$(psql $CONN_STRING -tc "select count(*) from tlds where type = $TTYPE")

for tld in $(curl ftp://rs.internic.net/domain/root.zone | awk -F\. ' { if (NF == 2) print tolower($1) } ' | awk '/./' | sort | uniq ) ; do

    echo "insert into tlds (alabel, type) values ('$tld', $TTYPE);" >> $TLD_TMP

done
psql $CONN_STRING -tf $TLD_TMP
TCOUNT2=$(psql $CONN_STRING -tc "select count(*) from tlds where type = $TTYPE")
TNEW=$((TCOUNT2 - TCOUNT1))
echo "Added $TNEW New gTLDs to database."
echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   datafiles_create_summary
#

# We need user with access to the database
a=$(@LIBEXEC@/check_user_perms -c)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD
RED='\033[0;31m'
LOG_FILE="dsc-extractor-xml-db.out"
NODE_NAME=""
SERVER_NAME=""
FORMAT_STRING="----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------"
FORMAT_SEP="|"

print_header () {
	echo $FORMAT_STRING
	echo " Server    $FORMAT_SEP Node      $FORMAT_SEP Last import started $FORMAT_SEP Last file processed $FORMAT_SEP Files found $FORMAT_SEP Files remaining $FORMAT_SEP Failures $FORMAT_SEP Last file processed                          $FORMAT_SEP Time      $FORMAT_SEP Error found in last"
	echo "           $FORMAT_SEP           $FORMAT_SEP                     $FORMAT_SEP at                  $FORMAT_SEP in incoming $FORMAT_SEP in import job   $FORMAT_SEP          $FORMAT_SEP                                              $FORMAT_SEP taken (s) $FORMAT_SEP 20 lines of file"
	echo $FORMAT_STRING
}

usage () {
	echo
	echo "Generate a processing report based on the information in the log files"
	echo 
	echo "Usage: $(basename $0) options"
	echo
	echo "Supported options:"
	echo "	-s Only report on the server with this name (default is all servers)"
	echo "	-n Only report on the node with this name (default is all nodes)"
	echo "	-c Output in csv format for import into spreadsheet application"
	echo "	-d Run in debug mode (set -x)"
	echo "	-h Show this help."
}

while getopts ":s:n:cdh" opt; do
    case $opt in
		n  ) NODE_NAME=$OPTARG ;;
		s  ) SERVER_NAME=$OPTARG ;;
		d  ) set -x ;;
		c  ) FORMAT_STRING=""
		     FORMAT_SEP=',';;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

TOTAL_IN_PROGRESS=0

if [ -z "${SERVER_NAME}" ] ; then
	SERVER_NAMES=$(psql $CONN_STRING -tc "select display_name from dsc.server")
else 
	SERVER_NAMES=$SERVER_NAME
fi

print_header

MIDNIGHT=$(date --date="00:00 today" +%s)

declare -a PROCESSING_NODES

for SERVER in $SERVER_NAMES ; do

	test -L $DATADIR/$SERVER && continue;
	test -d $DATADIR/$SERVER || continue;

	SERVER_ID=$(psql $CONN_STRING -tc "select id from dsc.server where display_name='${SERVER}'")

	if [ -z "${NODE_NAME}" ] ; then
		NODE_NAMES=$(psql $CONN_STRING -tc "select name from dsc.node where server_id=$SERVER_ID")
	else
		NODE_NAMES=$NODE_NAME
	fi
	COUNTER=0
	for NAME in $NODE_NAMES  ; do

		FILE_NAME=$DATADIR/$SERVER/$NAME/$LOG_FILE
		if [ ! -f "${FILE_NAME}" ] ; then
			continue
		fi

		if [ ! -d "$DATADIR/$SERVER/$NAME/incoming/" ] ; then
			continue
		fi

		if [ ! "$(ls -A $DATADIR/$SERVER/$NAME/incoming/)" ] ; then
			continue
		fi

		COUNTER=$(( COUNTER + 1 ))
		# Get the last date that import started:
		IMPORT_START_TIME=$(tac ${FILE_NAME} | grep -e 'UTC ----------------------$' -m1 | awk '{print $2, $3}')
		COMPLETED_LINE=$(tac ${FILE_NAME} | grep -e 'Completed' -m1 | awk '{print $1, $2, $5, $6, $8, $9}')
		IMPORT_LATEST_TIME=$(echo $COMPLETED_LINE | awk '{print $1, $2}')

		FOUND_FILES=$(find ${DATADIR}/${SERVER}/${NAME}/incoming/*/ -name \*.xml | wc -l)
		FILES_REMAINING_NUMBER=$(echo $COMPLETED_LINE | awk '{print $4}')
		FILES_FOUND_NUMBER=$(echo $FOUND_FILES | awk '{print $1}' )
		FILES_FAILED=$(echo $COMPLETED_LINE | sed 's/(//'  | awk '{print $5}')

		LAST_FILE_PATH=$(echo $COMPLETED_LINE | awk '{print $3}')
		LAST_FILE_NAME=$(basename "$LAST_FILE_PATH")
		LAST_FILE_UNIX_TIME="${LAST_FILE_NAME%.*.*}"
		if [ -z $LAST_FILE_UNIX_TIME ] ; then
			LAST_FILE=""
		else
			LAST_FILE=$(date -d @${LAST_FILE_UNIX_TIME} "+%Y-%m-%d %H:%M:%S")
		        LAST_FILE_START_TIME=$(tac ${FILE_NAME} |  grep -e "Starting:  ${LAST_FILE_PATH}"  -m 1 | awk '{print $1 " " $2}')
			LAST_FILE_FINISH_TIME=$(echo $COMPLETED_LINE | awk '{print $1 " " $2}')
			LAST_FILE_START_UNIX_TIME=$(date --date="${LAST_FILE_START_TIME}" +"%s")
			LAST_FILE_FINISH_UNIX_TIME=$(date --date="${LAST_FILE_FINISH_TIME}" +"%s")
			DURATION=$(( LAST_FILE_FINISH_UNIX_TIME - LAST_FILE_START_UNIX_TIME))
		fi
		(( TOTAL_FILES_FOUND = TOTAL_FILES_FOUND + FILES_FOUND_NUMBER ))

		ERR_COUNT=$(tail ${FILE_NAME} -n 20 | grep  -i -e 'Error' | grep  -v 'lock pid file' | wc -l)
		if [ $ERR_COUNT -ne 0 ] ; then
			ERROR_STR="YES"
		else
			ERROR_STR="NO"
		fi

		if [ $FILES_REMAINING_NUMBER -gt 0 ] && [ $ERROR_STR == "NO" ] ; then
			(( TOTAL_IN_PROGRESS = TOTAL_IN_PROGRESS + 1 ))
			(( TOTAL_PROCESSING_TIME = TOTAL_PROCESSING_TIME + DURATION ))
			PROCESSING_NODES+=" "${SERVER}/${NAME}
		fi
		
		if [ $LAST_FILE_UNIX_TIME -lt $MIDNIGHT ] ; then
			# color red
			printf '\033[0;31m%-10s %-1s %-9s %-1s %-16s %-1s %-19s %-1s %-11s %-1s %-15s %-1s %-8s %-1s %-19s %-22s %-1s %-9s %-1s %-15s\033[0m\n'  "$SERVER" $FORMAT_SEP  "$NAME" $FORMAT_SEP "${IMPORT_START_TIME}" $FORMAT_SEP "$IMPORT_LATEST_TIME" $FORMAT_SEP "$FOUND_FILES" $FORMAT_SEP "$FILES_REMAINING_NUMBER" $FORMAT_SEP  "$FILES_FAILED" $FORMAT_SEP "$LAST_FILE" "($LAST_FILE_NAME)" $FORMAT_SEP "$DURATION" $FORMAT_SEP "${ERROR_STR}"
		elif [ $FOUND_FILES -gt 720 ] ; then
			# color yellow
			printf '\033[0;33m%-10s %-1s %-9s %-1s %-16s %-1s %-19s %-1s %-11s %-1s %-15s %-1s %-8s %-1s %-19s %-22s %-1s %-9s %-1s %-15s\033[0m\n'  "$SERVER" $FORMAT_SEP  "$NAME" $FORMAT_SEP "${IMPORT_START_TIME}" $FORMAT_SEP "$IMPORT_LATEST_TIME" $FORMAT_SEP "$FOUND_FILES" $FORMAT_SEP "$FILES_REMAINING_NUMBER" $FORMAT_SEP  "$FILES_FAILED" $FORMAT_SEP "$LAST_FILE" "($LAST_FILE_NAME)" $FORMAT_SEP "$DURATION" $FORMAT_SEP "${ERROR_STR}"
		else
			# no color
			printf '%-10s %-1s %-9s %-1s %-16s %-1s %-19s %-1s %-11s %-1s %-15s %-1s %-8s %-1s %-19s %-22s %-1s %-9s %-1s %-15s\n'  "$SERVER" $FORMAT_SEP  "$NAME" $FORMAT_SEP "${IMPORT_START_TIME}" $FORMAT_SEP "$IMPORT_LATEST_TIME" $FORMAT_SEP "$FOUND_FILES" $FORMAT_SEP "$FILES_REMAINING_NUMBER" $FORMAT_SEP  "$FILES_FAILED" $FORMAT_SEP "$LAST_FILE" "($LAST_FILE_NAME)" $FORMAT_SEP "$DURATION" $FORMAT_SEP "${ERROR_STR}"
		fi

		if [ $COUNTER -eq 21 ] ; then
			print_header
			COUNTER=0
		fi
	done
done
print_header

echo "Time: " $(date "+%Y-%m-%d %H:%M:%S") " Total files remaining to import: " $TOTAL_FILES_FOUND
PERF=$(( TOTAL_PROCESSING_TIME / TOTAL_IN_PROGRESS ))
echo "Time: " $(date "+%Y-%m-%d %H:%M:%S") " Processors in progress: " ${TOTAL_IN_PROGRESS} " at approx " ${PERF} " seconds per datafile"
echo "List of nodes currently processing: $PROCESSING_NODES"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   datafiles_rm_empty_xml_dirs.bash
#

# We need user with access to the data directory
a=$(@LIBEXEC@/check_user_perms -fnc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a


DATE=$(date --date="- 7 day" +%Y-%m-%d)
INCOMING="incoming"

usage () {
	echo
	echo "Remove empty incoming xml directories older than a certain date (default is all the processed directories older than 7 days ago)"
	echo 
	echo "Usage: $(basename $0) options"
	echo
	echo "Supported options:"
	echo "	-d Date before which to remove empty xml directories (default: 7 days ago, format YYYY-MM-DD)"
	echo "	-h Show this help."
}

while getopts ":d:h" opt; do
    case $opt in
        d  ) DATE=$OPTARG ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

[ -z "$DATE" ] && usage && exit 1

# Validate the start date
if [[ ! $DATE =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]$ ]] ; then
		echo "Error: The start date is not in the required format of YYYY-MM-DD" 
		exit 1
fi
date "+%Y-%m-%d" --date="$DATE" >/dev/null 2>&1
is_valid=$?
if [ ${is_valid} -ne 0 ] ; then
	echo "Error: The start date is invalid" 
	exit 1
fi

# Date converted in seconds to be compared with processed directories date
DATE_SECONDS=$(date --utc --date "$DATE" +%s)

cd $DATADIR

for SERVER in * ; do
	test -L $SERVER && continue;
	test -d $SERVER || continue;
	cd $DATADIR/$SERVER
	for NODE in * ; do
		test -L $NODE && continue;
		test -d $NODE || continue;
		test -L $NODE/$DONE && continue;
        test -d $NODE/$DONE || continue;
        cd $DATADIR/$SERVER/$NODE/$INCOMING
		for MYDATE in * ; do
			test -L $MYDATE && continue;
			test -d $MYDATE || continue;
			date "+%Y-%m-%d" -d $MYDATE >/dev/null 2>&1
			is_valid=$?
			if [ ${is_valid} -ne 0 ] ; then
				continue;
			fi
			# Check if directory is older than DATE
			MYDATE_SECONDS=$(date --utc --date "$MYDATE" +%s)
			[ $MYDATE_SECONDS -ge $DATE_SECONDS ] && continue;
			# Remove empty directories
			my_count=$(ls -1 $MYDATE | wc -l)
			[ $my_count -eq 0 ] && rm -rf $MYDATE
		done
		cd $DATADIR/$SERVER
	done
	cd $DATADIR
done

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# We need user with access to the data directory
a=$(@LIBEXEC@/check_user_perms -fnc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a

usage () {
  echo
  echo "Pack old xml files already processed into a .tar package to store and archive."
  echo "The xml files are packed according to their date and node and server."
  echo "The filename has the following format done-PID-date.tar.bz2 (e.g. done-14143-2015-05-01.tar.bz2)."
  echo 
  echo "Usage: $(basename $0) options"
  echo
  echo "Supported options:"
  echo "  -h Show this help."
  exit 1
}

while getopts ":h" opt; do
    case $opt in
        h  ) usage ;;
        \? ) usage ;;
    esac
done

echo "$(date): $0 COMMENCED"

for i in $(find $DATADIR -maxdepth 3 -name "done" -type d) ; do
    for j in $(find $i -mindepth 1 -maxdepth 1 -type d) ; do
    short=$(echo $j | sed "s|$DATADIR||")
    DIR_FOUND=$(echo $short | awk -F\/ ' { print $5 } ')
    TODAY=$(date +%Y-%m-%d)
    if [ $DIR_FOUND == $TODAY ] ; then
      continue
    else
      echo "tar -cj --remove-files -f $i/done-$$-$DIR_FOUND.tar.bz2 $j"
      nice -n 19 tar -cj --remove-files -f $i/done-$$-$DIR_FOUND.tar.bz2 $j
    fi
  done
done

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   database_create
#

DB_NAME=hedgehog
DB_OWNER=hedgehog
DB_READER=www-data

usage () {
	echo
	echo "Create database and roles/users."
	echo
	echo "Usage: $(basename $0) options"
	echo
	echo "Supported options:"
	echo "  -o Owner of the database  (Default: $DB_OWNER)"
	echo "  -r Reader of the database (Default: $DB_READER) Typically the same user that the web server runs as."
	echo "  -n Name of the database   (Default: $DB_NAME)"
	echo "  -h Show this help"
	echo
	echo "In the 'database' section of hedgehog.yaml you should set the same values"
  
  exit 1
}

while getopts ":o:r:n:h" opt; do
    case $opt in
        o  ) DB_OWNER=$OPTARG ;;
        r  ) DB_READER=$OPTARG ;;
        n  ) DB_NAME=$OPTARG ;;
        h  ) usage ;;
        \? ) usage ;;
    esac
done

DB_EXISTS=$(psql -tlq | cut -d \| -f 1 | grep -w $DB_NAME)
[ "x$DB_EXISTS" != "x" ] && echo "The $DB_NAME Database exists. If you want to overwrite it then drop it manually and re-run this script" && exit 1

psql <<EOF
CREATE USER "$DB_OWNER";
CREATE ROLE hh_read;
CREATE ROLE hh_write;
CREATE DATABASE "$DB_NAME"
  WITH OWNER = "$DB_OWNER"
  ENCODING = 'UTF8'
  TABLESPACE = pg_default
  CONNECTION LIMIT = -1;
CREATE USER "$DB_READER";
GRANT hh_read TO "$DB_READER";
ALTER USER "$DB_READER" SET search_path TO dsc;
ALTER USER "$DB_OWNER" SET search_path TO dsc;
\connect "$DB_NAME"
CREATE LANGUAGE plpythonu;
CREATE SCHEMA dsc AUTHORIZATION "$DB_OWNER";
CREATE EXTENSION ip4r SCHEMA dsc;


CREATE or REPLACE FUNCTION dsc.ip2asn (addr dsc.ipaddress)
  RETURNS text
as \$\$

  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
  if rdata != None:
    result = rdata[0]['teamcymrulookup']
    if result != None:
      return result.split('|')[0].strip()
          
  plpy.info('Function ip2asn: No ASN found')
  return None
\$\$ LANGUAGE plpythonu;

CREATE or REPLACE FUNCTION dsc.ip2bgpprefix (addr dsc.ipaddress)
  RETURNS int
as \$\$

  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
  if rdata != None:
    result = rdata[0]['teamcymrulookup']
    if result != None:
      bgpprefix = result.split('|')[1].strip()
      return int(bgpprefix.split('/')[1])
          
  plpy.info('Function ip2bgpprefix: No BGP Prefix found')
  return None
\$\$ LANGUAGE plpythonu;

CREATE or REPLACE FUNCTION dsc.teamcymrulookup (addr dsc.ipaddress)
  RETURNS text
as \$\$
  from IPy import IP
  import getdns
  
  ''' Sanity Check '''
  if addr is None:
    plpy.error('Function teamcymrulookup: Address passed in was null')
    return None
    
  ''' Create a IPy IP object '''
  ip = IP(addr)

  ''' reverse the address '''
  if ip.version() == 4:
    s = ip.strFullsize()
    s = s.split('.')
    s.reverse()
    s = '.'.join(s)
    qname = "%s.origin.asn.cymru.com." % s
  elif ip.version() == 6:
    s = '%032x' % ip.int()
    s = list(s)
    s.reverse()
    s = '.'.join(s)
    qname = "%s.origin6.asn.cymru.com." % s 
  else:
    plpy.error('Function teamcymrulookup: Unknown IP address type.')
    return None
    
  ''' Store/retrieve the context in/from the global dict. '''
  if 'teamcymrulookupctx' in GD.keys():
      ctx = GD['teamcymrulookupctx']
  else:
      ctx = getdns.Context()
      ctx.resolution_type = getdns.RESOLUTION_STUB
      GD['teamcymrulookupctx'] = ctx
  
  try:
    results = ctx.general(name=qname, request_type=getdns.RRTYPE_TXT, extensions={})
  except getdns.error as e:
    plpy.error('Function teamcymrulookup: GetDNS Error: %s' % str(e))
    return None
  status = results.status
  if status == getdns.RESPSTATUS_GOOD:
    for reply in results.replies_tree:
      answers = reply['answer']
      for answer in answers:
        if answer['type'] == getdns.RRTYPE_TXT:
          rdata = answer['rdata']
          return rdata['txt_strings'][0]
          
  plpy.info('Function teamcymrulookup: No data returned from DNS query')
  return None
\$\$ LANGUAGE plpythonu;

CREATE or REPLACE FUNCTION dsc.iptruncate (addr dsc.ipaddress)
  RETURNS dsc.iprange
as \$\$
BEGIN
  CASE family(addr)
    WHEN 4 then RETURN addr / 8;
    WHEN 6 then RETURN addr / 32;
  END CASE;
END
\$\$ LANGUAGE plpgsql;

CREATE or REPLACE FUNCTION dsc.alabel2ulabel (alabel text)
  RETURNS text
as \$\$
  
  import getdns
  
  ''' Sanity Check '''
  if alabel is None:
    plpy.error('Function alabel2ulabel: a-label passed in was null')
    return None
  
  ulabel = getdns.alabel_to_ulabel(alabel)
  return ulabel
\$\$ LANGUAGE plpythonu;
EOF

echo
echo "Please add $DB_OWNER and $DB_READER to pg_hba.conf, set passwords etc."
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#

# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   plotcache_generate_cached_plots
#

# We need user with access to the plotcache directory
a=$(@LIBEXEC@/check_user_perms -pc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

FROM=$(date --date="- 1 day" +%Y-%m-%dT00:00)
TO=$(date --date="- 1 day" +%Y-%m-%dT23:59)
SQ="'"

usage () {
    echo
    echo "Create cached plots for each chosen period (day/week/month) between Start and End range of time"
    echo "You must add at least one option for which plot window to generate [-D|-W|-M]"
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -s Start date and time for which to create cached plots (default: \"${FROM}\")"
    echo "  -e End date and time for which to create cached plots (default: \"${TO}\")"
    echo "  -D Generate daily cached plots"
    echo "  -W Generate weekly cached plots"
    echo "  -M Generate monthly cached plots"
    echo "  -h Show this help."
}

while getopts ":s:e:DWMh" opt; do
    case $opt in
        s  ) FROM=$OPTARG ;;
        e  ) TO=$OPTARG ;;
        D  ) DAY=1 ;;
        W  ) WEEK=1 ;;
        M  ) MONTH=1 ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

# Validate the start date
if [[ ! $FROM =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]$ ]] ; then
		echo "Error: The start date is not in the required format of YYYY-MM-DDThh:mm" 
		exit 1
fi
date "+%Y-%m-%dT%H:%M" --date="${FROM}" >/dev/null 2>&1
is_valid=$?
if [ ${is_valid} -ne 0 ] ; then
	echo "Error: The start date is invalid" 
	exit 1
fi

# Validate the end date
if [[ ! $TO =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]$ ]] ; then
		echo "Error: The end date is not in the required format of YYYY-MM-DDThh:mm" 
		exit 1
fi
date "+%Y-%m-%dT%H:%M" --date="${TO}" >/dev/null 2>&1
is_valid=$?
if [ ${is_valid} -ne 0 ] ; then
	echo "Error: The end date is invalid" 
	exit 1
fi

# Check that Start date is earlier than End date
FROM_seconds=$(date --utc --date "${FROM}" +%s)
TO_seconds=$(date --utc --date "${TO}" +%s)
if [ ${FROM_seconds} -ge ${TO_seconds} ] ; then
  echo "Error: Start date must be earlier than End date to generate plots"
  exit 1
fi

if [[ -z "$DAY" && -z "$WEEK" && -z "$MONTH" ]] ; then
  usage
  exit 1
fi

echo "INFO: $(find $PLOT_CACHE_DIR -type f | wc -l) cached plots"
echo "SYSMON INFO: $(du -c $PLOT_CACHE_DIR | grep total) disk usage for $PLOT_CACHE_DIR"

# TODO: do not hard code this path
URL="localhost/hedgehog/brew/hedgehog.brew"
TYPE="generatePlot"
declare -a STOPS=($(seq $(date +%s -d ${TO}) -86400 $(date +%s -d ${FROM})))
INTERACTIVE="0 1"
PLOTIDS=$(psql $CONN_STRING -tc "select id from dsc.plot where name in (select name from dsc.visible_plot) ORDER BY id DESC;")
SERVERNMS=$(psql $CONN_STRING -tc "SELECT display_name FROM dsc.server;")
NDARR=-1 # all nodes

# Call brew script in order to cache the default plots
for SVRNM in $SERVERNMS ; do
    SVRID=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server WHERE display_name=${SQ}${SVRNM}${SQ};" | tr -d [[:space:]])
    for STOP in ${STOPS[@]} ; do
        STOP=$(date --date @${STOP} +%Y-%m-%dT%R)
        STARTS=()
        [ -n "$DAY" ] && STARTS+=($(date --date="${STOP}" +%Y-%m-%dT00:00))
        [ -n "$WEEK" ] && STARTS+=($(date --date="${STOP} 1 week ago + 1 day" +%Y-%m-%dT00:00))
        [ -n "$MONTH" ] && STARTS+=($(date --date="${STOP} 1 month ago + 1 day" +%Y-%m-%dT00:00))
        for START in ${STARTS[@]} ; do
            for GVIS in $INTERACTIVE ; do
                for PLTID in $PLOTIDS ; do
                    echo $(date) ": curl --max-time 300 --retry 1 --get \"${URL}?type=${TYPE}&start=${START}&stop=${STOP}&gvis=${GVIS}&pltid=${PLTID}&svrnm=${SVRNM}&svrid=${SVRID}&ndarr=${NDARR}\""
                    curl -s -o /dev/null --max-time 300 --retry 1 --get "${URL}?type=${TYPE}&start=${START}&stop=${STOP}&gvis=${GVIS}&pltid=${PLTID}&svrnm=${SVRNM}&svrid=${SVRID}&ndarr=${NDARR}" 2>&1
                done
            done
        done
    done
done

echo "INFO: $(find $PLOT_CACHE_DIR -type f | wc -l) cached plots"
echo "SYSMON INFO: $(du -c $PLOT_CACHE_DIR | grep total) disk usage for $PLOT_CACHE_DIR"
echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   plotcache_rm_cached_plots
#

# We need user with access to the plotcache directory
a=$(@LIBEXEC@/check_user_perms -pc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

REMOVE_ALL=0

usage () {
    echo
    echo "This script is interactive if the -a option is not selected."
    echo "You will be asked to select which cached plot types you'd like to remove"
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -a Remove all cached plots"
    echo "  -h Show this help."
}

while getopts ":ah" opt; do
    case $opt in
        a  ) REMOVE_ALL=1 ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

if [ $REMOVE_ALL -eq 1 ] ; then
	echo "Removing all cached plots from $PLOT_CACHE_DIR"
	rm -rf $PLOT_CACHE_DIR/*/*/*.png
	rm -rf $PLOT_CACHE_DIR/*/*/*.html
	echo "Done."
	exit
fi

# Get list of servers and plots from database
SERVERS=$(psql $CONN_STRING -tc "SELECT display_name FROM dsc.server;")
PLOTS=$(psql $CONN_STRING -tc "SELECT name FROM dsc.visible_plot;")
PLOTS=(${PLOTS// / }) # convert string to an array

while true ; do
    if [ ${#PLOTS[@]} -eq 0 ] ; then
        exit
    fi
    echo "Please select which cached plot type you'd like to clear or q to exit:"
    echo
    n=0
    for plot in ${PLOTS[@]} ; do
        let n=n+1
        echo "    ${n}. ${plot}"
    done
    echo
    echo -n "Enter selection or q to exit: "
    read c
    if [ "$c" == "Q" ] || [ "$c" == "q" ] ; then
        exit 1
    elif [[ $c -ge 1 ]] && [[ $c -le $n ]] ; then
        let c=c-1
        selection=${PLOTS[${c}]}
        PLOTS=(${PLOTS[@]:0:$c} ${PLOTS[@]:$(($c + 1))}) # remove selected plot from PLOTS
        for server in $SERVERS ; do
            echo "rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.png"
            rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.png
            echo "rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.html"
            rm -rf $PLOT_CACHE_DIR/${server}/${selection}/*.html
            echo "Removed plots for ${selection}."
        done
    else
        echo "invalid selection - please try again or exit by typing 'q'"
    fi
	echo "Do you want to remove another plot?"
done

echo "$(date): $0 COMPLETED"
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# 
# File:   rssac_generate_reports
#

#TODO(asap): Add a flag to allow the user to specify a single server to run the report for

# eval $(@LIBEXEC@/conf_read)

a=$(@LIBEXEC@/check_user_perms -pgc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

DEFAULT_DAY=$(date --date="- 7 day" +%Y-%m-%d)
START=$(date --date="${DEFAULT_DAY}" +%Y-%m-%dT00:00)
INPUT_START=""

usage () {
    echo
    echo "Create RSSAC reports for a specified 24 hour period"
    echo "(Requires data to have been processed using the database_process_rssac_data script)"
    echo
    echo "Usage: $(basename $0) options"
    echo
    echo "Supported options:"
    echo "  -d Date on which to create RSSAC reports (default is 7 days ago: \"${DEFAULT_DAY}\", format: YYYY-MM-DD)"
    echo "  -h Show this help"
}

while getopts ":d:h" opt; do
    case $opt in
        d  ) INPUT_START=$OPTARG ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

echo "$(date): $0 COMMENCED"

# Validate the input date
if [ ! -z "${INPUT_START}" ] ; then
	if [[ ! $INPUT_START =~ ^[0-9]{4}-[0-1][0-9]-[0-3][0-9]$ ]] ; then
			echo "Error: The start date is not in the required format of YYYY-MM-DD" 
			exit 1
	fi
	date "+%Y-%m-%d" --date="${INPUT_START}" >/dev/null 2>&1
	is_valid=$?
	if [ ${is_valid} -ne 0 ] ; then
		echo "Error: The start date is invalid" 
		exit 1
	fi
	START=$(date --date="${INPUT_START}" +%Y-%m-%dT00:00)
        DEFAULT_DAY=$(date --date="${INPUT_START}" +%Y-%m-%d)
fi

STOP=$(date --date="${START}" +%Y-%m-%dT23:59)

SQ="'"
CURL_ARGS="--max-time 300 --retry 1 --get"
# TODO: do not hardcode this path
URL="localhost/hedgehog/brew/hedgehog.brew"
TOP_OUTPUTDIR="$RSSAC_DIR/$(date --date=${START} +%Y)/$(date --date=${START} +%m)"
TYPES="generateYaml generatePlot"
GVIS=0
# Specify some strings for special handling
TRAFFIC_SIZE_SMALL="traffic_sizes_small"
TRAFFIC_SIZE_BIG="traffic_sizes_big"
TRAFFIC_SIZE_REP_NAME="traffic-sizes"
TRAFFIC_VOLUME_DIFF="traffic_volume_difference"

echo
echo "Creating RSSAC reports from ${START} to ${STOP} in ${TOP_OUTPUTDIR}" 
echo

PLOTIDS=$(psql $CONN_STRING -tc "SELECT id FROM dsc.plot where ddcategory='9.RSSAC' and name != '$TRAFFIC_VOLUME_DIFF' ORDER BY id DESC;")
SERVERNMS=$(psql $CONN_STRING -tc "SELECT display_name FROM dsc.server;")
NDARR=-1 # all nodes

for SVRNM in $SERVERNMS ; do
    SVRID=$(psql $CONN_STRING -tc "SELECT id FROM dsc.server WHERE display_name=${SQ}${SVRNM}${SQ};" | tr -d [[:space:]])
    SVRNM_LC="$(echo ${SVRNM} | tr '[:upper:]' '[:lower:]')"

    for TYPE in $TYPES ; do
        EXT="yaml"
        if [ "${TYPE}" == "generatePlot" ] ; then
			EXT="png"
		fi
		
        for PLTID in $PLOTIDS ; do
			PLTNM=$(psql $CONN_STRING -tc "SELECT name FROM dsc.plot where id=${PLTID};" | sed 's/[[:space:]]//g')
			# Clear out the file in the plot cache to make sure we get the most up to date plots in the reports
			rm -rf $RSSAC_DIR/${SVRNM}/${PLTNM}/${START}-${STOP}-all*.png >/dev/null 2>&1
			# Generate the file. Yaml gets written to the rssac directory, plots go into the plotcache
			echo "*** Generating '${EXT}' for: " ${PLTNM}
			CURL_URL="${URL}?type=${TYPE}&start=${START}&stop=${STOP}&gvis=${GVIS}&pltid=${PLTID}&svrnm=${SVRNM}&svrid=${SVRID}&ndarr=${NDARR}"
            echo $(date) ": curl $CURL_ARGS \"$CURL_URL\""
            RESULT=$(curl $CURL_ARGS "$CURL_URL" 2>&1)
            echo ${RESULT}
			echo
			
            if [ "${TYPE}" == "generatePlot" ] ; then
				# Retrieve the plot from the plot cache
                PLTFILE=$(echo ${RESULT} | sed 's/^.* src="\([^"?]*\).*/\1/' | sed "s:../plotcache:$PLOT_CACHE_DIR:" | sed 's/-log//' | sed 's/-stack//')

                if [ "${PLTFILE}" != "plots/no_graph.png" ] && [ "${PLTFILE}" != "plots/no_results.png" ] && [ "${PLTFILE}" != "plots/no_connection.png" ] ; then                   
					# Report names are the same as plot names but use dash instead of underscore
					# TODO: use $PLTNM instead of redoing the query then substitute _ with -
					REP_NAME=$(psql $CONN_STRING -tc "SELECT name FROM dsc.plot where id=${PLTID};" | sed 's/_/-/g' | sed 's/[[:space:]]//g')
					# Special handling. The traffic size data has 2 plots but both need to go into the same directory
					PLT_DIR_NM=${REP_NAME}
					if [ "${PLTNM}" == "${TRAFFIC_SIZE_SMALL}" ] || [ "${PLTNM}" == "${TRAFFIC_SIZE_BIG}" ] ; then
						PLT_DIR_NM="${TRAFFIC_SIZE_REP_NAME}"
					fi
                    OUTPUTDIR="${TOP_OUTPUTDIR}/${PLT_DIR_NM}/"
                    OUTPUTFILE="${SVRNM_LC}-`date --date=${START} +%Y%m%d`-${REP_NAME}.png"
                    cp ${PLTFILE} ${OUTPUTDIR}${OUTPUTFILE}
                fi
            fi
        done
    done
done
echo "$(date): $0 COMPLETED"
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* Take an IPv[46] address and look it up in the Team Cymru IP to ASN DNS service
 * See http://www.team-cymru.org/IP-ASN-mapping.html
 */

CREATE or REPLACE FUNCTION dsc.ip2asn (addr dsc.ipaddress)
  RETURNS text
as $$

  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
  if rdata != None:
    result = rdata[0]['teamcymrulookup']
    if result != None:
      return result.split('|')[0].strip()
          
  plpy.info('Function ip2asn: No ASN found')
  return None
$$ LANGUAGE plpythonu;

CREATE or REPLACE FUNCTION dsc.ip2bgpprefix (addr dsc.ipaddress)
  RETURNS int
as $$

  rdata = plpy.execute("select dsc.teamcymrulookup('" + addr + "')")
  if rdata != None:
    result = rdata[0]['teamcymrulookup']
    if result != None:
      bgpprefix = result.split('|')[1].strip()
      return int(bgpprefix.split('/')[1])
          
  plpy.info('Function ip2bgpprefix: No BGP Prefix found')
  return None
$$ LANGUAGE plpythonu;

CREATE or REPLACE FUNCTION dsc.teamcymrulookup (addr dsc.ipaddress)
  RETURNS text
as $$
  from IPy import IP
  import getdns
  
  ''' Sanity Check '''
  if addr is None:
    plpy.error('Function teamcymrulookup: Address passed in was null')
    return None
    
  ''' Create a IPy IP object '''
  ip = IP(addr)

  ''' reverse the address '''
  if ip.version() == 4:
    s = ip.strFullsize()
    s = s.split('.')
    s.reverse()
    s = '.'.join(s)
    qname = "%s.origin.asn.cymru.com." % s
  elif ip.version() == 6:
    s = '%032x' % ip.int()
    s = list(s)
    s.reverse()
    s = '.'.join(s)
    qname = "%s.origin6.asn.cymru.com." % s 
  else:
    plpy.error('Function teamcymrulookup: Unknown IP address type.')
    return None
    
  ''' Store/retrieve the context in/from the global dict. '''
  if 'getdnsctx' in GD.keys():
      ctx = GD['getdnsctx']
  else:
      ctx = getdns.Context()
      ctx.resolution_type = getdns.RESOLUTION_STUB
      GD['getdnsctx'] = ctx
  
  try:
    results = ctx.general(name=qname, request_type=getdns.RRTYPE_TXT, extensions={})
  except getdns.error as e:
    plpy.error('Function teamcymrulookup: GetDNS Error: %s' % str(e))
    return None
  status = results.status
  if status == getdns.RESPSTATUS_GOOD:
    for reply in results.replies_tree:
      answers = reply['answer']
      for answer in answers:
        if answer['type'] == getdns.RRTYPE_TXT:
          rdata = answer['rdata']
          return rdata['txt_strings'][0]
          
  plpy.info('Function teamcymrulookup: No data returned from DNS query')
  return None
$$ LANGUAGE plpythonu;

CREATE or REPLACE FUNCTION dsc.iptruncate (addr dsc.ipaddress)
  RETURNS dsc.iprange
as $$
BEGIN
  CASE family(addr)
    WHEN 4 then RETURN addr / 8;
    WHEN 6 then RETURN addr / 32;
  END CASE;
END
$$ LANGUAGE plpgsql;

CREATE or REPLACE FUNCTION dsc.alabel2ulabel (alabel text)
  RETURNS text
as $$
  
  import getdns
  
  ''' Sanity Check '''
  if alabel is None:
    plpy.error('Function alabel2ulabel: a-label passed in was null')
    return None
  
  ulabel = getdns.alabel_to_ulabel(alabel)
  return ulabel
$$ LANGUAGE plpythonu;
/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

-----------------
--  DSC SCHEMA --
-----------------

--
-- Name: version; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.version (
    version integer
);

--
-- Name: internal_version; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.internal_version (
    serial integer,
    script character varying(255),
    description character varying(255),
    applied timestamp with time zone
);

--
-- Name: server; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.server
(
  id serial NOT NULL,
  name character varying(255) NOT NULL,
  display_name character varying(255)  NOT NULL,
  description character varying(255)
)
WITH (
  OIDS=FALSE
);

--
-- Name: node; Type: TABLE; Schema: dsc; 
--
--TODO(asap): Just have server_id, name and group
CREATE TABLE dsc.node
(
  id serial NOT NULL,
  server_id integer NOT NULL,
  name character varying(255) NOT NULL,
  city character varying(255),
  state character varying(255),
  country character varying(255),
  region character varying(255) NOT NULL
)
WITH (
  OIDS=FALSE
);

--
-- Name: dataset; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.dataset
(
  id serial NOT NULL,
  name character varying(255) NOT NULL
)
WITH (
  OIDS=FALSE
);

--
-- Name: visible_plot; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.visible_plot
(
  id serial NOT NULL, 
  name character varying(255) NOT NULL	,
  display_category character varying(255),
  display_name character varying(255),
  title character varying(255),
  description character varying(255)
)
WITH (
  OIDS=FALSE
);

--
-- Name: dataset_visible_plot; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.dataset_visible_plot
(
  visible_plot_id integer NOT NULL,
  dataset_id integer NOT NULL
)
WITH (
  OIDS=FALSE
);

--
-- Name: plot; Type: VIEW; Schema: dsc; 
-- This view is used for backwards compatibility and to provide an interface
-- into the new dataset and visible plots tables since there is now a 
-- many to many relationship between them. When the front end code is 
-- refactored it can be replaced by a cleaner view!
--
CREATE VIEW dsc.plot AS
(
	SELECT ROW_NUMBER() OVER(ORDER BY name) AS id, * FROM
	(
		SELECT
			vp.name,
			vp.display_category AS ddcategory,
			vp.display_name AS ddname,
			vp.title,
			vp.description,
			(SELECT dataset_id FROM dataset_visible_plot WHERE dsc.dataset_visible_plot.visible_plot_id=vp.id LIMIT 1) AS plot_id,
			(SELECT ARRAY(SELECT dataset_id FROM dataset_visible_plot WHERE dsc.dataset_visible_plot.visible_plot_id=vp.id)) AS dataset_id
		FROM
			dsc.visible_plot AS vp
		UNION
		SELECT
			name,
			'' AS ddcategory,
			'' AS ddname,
			'' AS title,
			'' AS description,
			id AS plot_id,
			ARRAY[id] AS dataset_id
		FROM
			dataset
		WHERE
			dsc.dataset.name NOT IN (SELECT name FROM dsc.visible_plot)
	)
	AS x
);

--
-- Name: query_classification; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.query_classification (
    id serial NOT NULL,
    name character varying(255),
    title character varying(255)
);

--
-- Name: iana_lookup; Type: TABLE; Schema: dsc; 
--
CREATE TABLE dsc.iana_lookup
(
  registry character varying(255)  NOT NULL,
  value integer NOT NULL,
  name character varying(255) NOT NULL,
  description character varying(255)
)
WITH (
  OIDS=FALSE
);

--
-- Name: data; Type: TABLE; Schema: dsc; 
--
--TODO(refactor): Strictly don't need the server_id here. 
CREATE TABLE dsc.data
(
  starttime timestamp with time zone NOT NULL,
  server_id integer NOT NULL,
  node_id integer NOT NULL,
  plot_id integer NOT NULL,
  key1 character varying(255) NOT NULL,
  key2 character varying(255) NOT NULL,
  value integer NOT NULL
)
WITH (
  OIDS=FALSE
);

--
-- Name: pk_server, uniq_server; Type: CONSTRAINT; Schema: dsc; 
--
ALTER TABLE ONLY dsc.server
  ADD CONSTRAINT pk_server PRIMARY KEY (id),
  ADD CONSTRAINT uniq_server UNIQUE (name);

--
-- Name: pk_nodes, fk_nodes_servers, uniq_nodes; Type: CONSTRAINT; Schema: dsc; 
--
ALTER TABLE ONLY dsc.node
  ADD CONSTRAINT pk_node PRIMARY KEY (id),
  ADD CONSTRAINT fk_node_server FOREIGN KEY (server_id) REFERENCES dsc.server (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION,
  ADD CONSTRAINT uniq_node UNIQUE(server_id, name);

--
-- Name: pk_dataset, uniq_dataset_name; Type: CONSTRAINT; Schema: dsc; 
--
ALTER TABLE ONLY dsc.dataset
  ADD CONSTRAINT pk_dataset PRIMARY KEY (id),
  ADD CONSTRAINT uniq_dataset_name UNIQUE (name);

--
-- Name: pk_visible_plot, uniq_visible_plot_name; Type: CONSTRAINT; Schema: dsc; 
--
ALTER TABLE ONLY dsc.visible_plot
  ADD CONSTRAINT pk_visible_plot PRIMARY KEY (id),
  ADD CONSTRAINT uniq_visible_plot_name UNIQUE (name);


--
-- Name: pk_il; Type: CONSTRAINT; Schema: dsc; 
--
ALTER TABLE ONLY dsc.iana_lookup
  ADD CONSTRAINT pk_il PRIMARY KEY (registry , value , name );

--
-- Name: pk_data, fk_data_node, fk_data_plot; Type: CONSTRAINT; Schema: dsc; 
--
ALTER TABLE dsc.data
  ADD CONSTRAINT pk_data PRIMARY KEY (server_id, node_id, plot_id, starttime, key1, key2),
  ADD CONSTRAINT fk_data_node FOREIGN KEY (node_id) REFERENCES dsc.node (id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION,
  ADD CONSTRAINT fk_data_plot FOREIGN KEY (plot_id) REFERENCES dsc.dataset(id) MATCH SIMPLE ON UPDATE NO ACTION ON DELETE NO ACTION;

/*
--
-- Name: idx_data; Type: INDEX; Schema: dsc; 
--
CREATE INDEX idx_data
  ON dsc.data
  USING btree
  (plot_id COLLATE pg_catalog."default" );
*/

--
-- Name: fn_dsc_data_insert_trigger; Type: FUNCTION; Schema: dsc; 
--
CREATE OR REPLACE FUNCTION dsc.fn_dsc_data_insert_trigger()
RETURNS TRIGGER AS $$
DECLARE
    server TEXT;
    plot TEXT;
    tbl_nm TEXT;
    ins_sql TEXT;
BEGIN
    SELECT name INTO STRICT server FROM dsc.server WHERE id = new.server_id;
    SELECT name INTO STRICT plot FROM dsc.dataset WHERE id = new.plot_id;
    tbl_nm := 'data_' || server || '_' || plot || '_' || to_char(NEW.starttime, 'YYYY_MM');
    ins_sql := 
        'INSERT INTO dsc.'|| tbl_nm
        || ' (starttime, server_id, node_id, plot_id, key1, key2, value) VALUES'
        || ' ('
        || quote_literal(NEW.starttime) || ','
        || NEW.server_id || ','
        || NEW.node_id || ','
        || NEW.plot_id || ','
        || quote_literal(NEW.key1) || ','
        || quote_literal(NEW.key2) || ','
        || NEW.value
        || ')'
        ;
    EXECUTE ins_sql; 
    RETURN NULL;
EXCEPTION
    WHEN undefined_table THEN
        RAISE WARNING 'dsc data has not been inserted due to undefined partition --> %', tbl_nm;
        RETURN NULL;
END
$$
LANGUAGE 'plpgsql' ;

--
-- Name: dsc_data_insert_trigger; Type: TRIGGER; Schema: dsc; 
--
CREATE TRIGGER dsc_data_insert_trigger
    BEFORE INSERT ON dsc.data
    FOR EACH ROW EXECUTE PROCEDURE dsc.fn_dsc_data_insert_trigger();

--
-- Name: unique_source_summary_function(integer, integer, timestamp with time zone, timestamp with time zone, text); Type: FUNCTION; Schema: dsc; 
--
CREATE FUNCTION unique_source_summary_function(integer, integer, timestamp with time zone, timestamp with time zone, text) RETURNS TABLE(x character varying, y bigint)
    LANGUAGE plpgsql
    AS $_$
BEGIN
return query SELECT 
				key1 as x, 
				count(key2) AS y 
				FROM dsc.data as d 
				WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) 
				GROUP BY x 
			UNION 
			SELECT 
				'IPv6/64' as x, 
				count(*) as y from (SELECT substring(key2 FROM '(^([0-9a-f]{1,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}))') as subnet 
				FROM dsc.data as d 
				WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1='IPv6' AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY subnet) AS sq 
				ORDER BY y DESC;
END
$_$;

/* 
 * Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
 * 
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, you can obtain one at https://mozilla.org/MPL/2.0/.
 */

/*
 * Developed by Sinodun IT (www.sinodun.com)
 */

/* Both are bumped from 2.0 version by 1 since DDL 10 is not run on fresh install.
   This is because it requires super user priviledges so is done
   as part of database_create. */
INSERT INTO dsc.version (version) VALUES 
    (8);

INSERT INTO dsc.internal_version ( serial, script, description, applied ) VALUES
    (10, '-', 'DB creation', now() );

INSERT INTO dataset (name) VALUES ('certain_qnames_vs_qtype');
INSERT INTO dataset (name) VALUES ('chaos_types_and_names');
INSERT INTO dataset (name) VALUES ('client_addr_vs_rcode_accum');
INSERT INTO dataset (name) VALUES ('client_subnet2_accum');
INSERT INTO dataset (name) VALUES ('client_subnet2_count');
INSERT INTO dataset (name) VALUES ('client_subnet2_trace');
INSERT INTO dataset (name) VALUES ('client_subnet_accum');
INSERT INTO dataset (name) VALUES ('client_subnet_count');
INSERT INTO dataset (name) VALUES ('direction_vs_ipproto');
INSERT INTO dataset (name) VALUES ('dns_ip_version');
INSERT INTO dataset (name) VALUES ('dns_ip_version_vs_qtype');
INSERT INTO dataset (name) VALUES ('dnssec_qtype');
INSERT INTO dataset (name) VALUES ('do_bit');
INSERT INTO dataset (name) VALUES ('edns_version');
INSERT INTO dataset (name) VALUES ('idn_qname');
INSERT INTO dataset (name) VALUES ('idn_vs_tld');
INSERT INTO dataset (name) VALUES ('ipv6_rsn_abusers_accum');
INSERT INTO dataset (name) VALUES ('ipv6_rsn_abusers_count');
INSERT INTO dataset (name) VALUES ('opcode');
INSERT INTO dataset (name) VALUES ('pcap_stats');
INSERT INTO dataset (name) VALUES ('qtype');
INSERT INTO dataset (name) VALUES ('qtype_vs_qnamelen');
INSERT INTO dataset (name) VALUES ('qtype_vs_tld');
INSERT INTO dataset (name) VALUES ('rcode');
INSERT INTO dataset (name) VALUES ('rcode_vs_replylen');
INSERT INTO dataset (name) VALUES ('rd_bit');
INSERT INTO dataset (name) VALUES ('traffic_sizes_queries');
INSERT INTO dataset (name) VALUES ('traffic_sizes_responses');
INSERT INTO dataset (name) VALUES ('traffic_volume_queries');
INSERT INTO dataset (name) VALUES ('traffic_volume_responses');
INSERT INTO dataset (name) VALUES ('transport_vs_qtype');
INSERT INTO dataset (name) VALUES ('unique_sources');
INSERT INTO dataset (name) VALUES ('unique_sources_raw');

INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('by_node', '1.Node Statistics', 'By node', 'Queries by node');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('chaos_types_and_names', '2.Query Attributes', 'CHAOS queries', 'CHAOS queries');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('do_bit', '2.Query Attributes', 'DO bit', 'Queries with the DNSSEC OK bit set');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('edns_version', '2.Query Attributes', 'EDNS version', 'EDNS version');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('idn_qname', '2.Query Attributes', 'IDN qnames', 'Queries containing internationalized qnames');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('opcode', '2.Query Attributes', 'OPCODE', 'Breakdown of OPCODE values, other than QUERY');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rd_bit', '2.Query Attributes', 'RD bit', 'Queries with Recursion Desired bit set');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('qtype', '3.QTYPE', 'DNS queries by QTYPE', 'QTYPE values in DNS queries');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('dnssec_qtype', '3.QTYPE', 'DNSSEC queries by QTYPE', 'QTYPE values in DNSSEC queries');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('certain_qnames_vs_qtype', '3.QTYPE', 'Popular query names by QTYPE', 'Queries for the name localhost and anything under root-servers.net.');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('qtype_vs_tld', '3.QTYPE', 'QTYPE for most popular TLDs', 'QTYPE values for most popular TLDs queried');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('qtype_vs_qnamelen', '3.QTYPE', 'Query name lengths by QTYPE', 'Query name lengths showing QTYPE');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode', '4.RCODE', 'Replies by RCODE', 'RCODE values in DNS replies');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode_vs_replylen', '4.RCODE', 'Reply lengths by RCODE (<1000 bytes)', 'Size of DNS replies showing RCODE for message size below 1000 bytes');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode_vs_replylen_big', '4.RCODE', 'Reply lengths by RCODE (>=1000 bytes)', 'Size of DNS replies showing RCODE for message size above 1000 bytes');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('dns_ip_version', '5.IP Protocol', 'IP version', 'IP version carrying DNS queries');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('dns_ip_version_vs_qtype', '5.IP Protocol', 'Queries by IP version and QTYPE', 'QTYPE by IP version');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('direction_vs_ipproto', '5.IP Protocol', 'Received packets by IP protocol', 'Recieved Packets by IP protocol');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('transport_vs_qtype', '5.IP Protocol', 'Transports carrying DNS queries', 'Transports carrying DNS queries');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('pcap_stats', '6.PCAP Statistics', 'PCAP Statistics', 'Packet capture statistics');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet_accum', '7.Client Subnet Statistics', 'Busiest client subnets', 'Busiest client subnets (IPv4/8 or IPv6/32)');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_addr_vs_rcode_accum', '7.Client Subnet Statistics', 'RCODE by client subnet', 'Busiest client subnets (IPv4/8 or IPv6/32) showing RCODE');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('ipv6_rsn_abusers_accum', '7.Client Subnet Statistics', 'Root abusers', 'Clients sending excessive root-servers.net queries');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet2_accum', '8.Classification', 'Query classification by client subnet (accum)', 'Query classification by client subnet (IPv4/8 or IPv6/32)');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet2_count', '8.Classification', 'Query classification by client subnet (count)', 'Count of client subnet sending each query classification');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('client_subnet2_trace', '8.Classification', 'Query classifications', 'Queries by classification');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('rcode_volume', '9.RSSAC', 'RCODE volume', 'Count of rcodes in responses');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_sizes_small', '9.RSSAC', 'Traffic sizes (<1000 bytes)', 'Histogram of DNS query and response sizes by transport for message size below 1000 bytes');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_sizes_big', '9.RSSAC', 'Traffic sizes (>=1000 bytes)', 'Histogram of DNS query and response sizes by transport for message size above 1000 bytes');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_volume', '9.RSSAC', 'Traffic volume', 'The number of queries and responses by transport and IP version');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('unique_sources', '9.RSSAC', 'Unique sources', 'The number of unique source addresses');
INSERT INTO visible_plot (name, display_category, display_name, title) VALUES ('traffic_volume_difference', '9.RSSAC', 'Traffic volume difference', 'The differences between queries and responses by transport and IP version');


INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='by_node'), (SELECT id FROM dataset WHERE name='qtype'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='chaos_types_and_names'), (SELECT id FROM dataset WHERE name='chaos_types_and_names'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='do_bit'), (SELECT id FROM dataset WHERE name='do_bit'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='edns_version'), (SELECT id FROM dataset WHERE name='edns_version'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='idn_qname'), (SELECT id FROM dataset WHERE name='idn_qname'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='opcode'), (SELECT id FROM dataset WHERE name='opcode'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rd_bit'), (SELECT id FROM dataset WHERE name='rd_bit'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='qtype'), (SELECT id FROM dataset WHERE name='qtype'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='dnssec_qtype'), (SELECT id FROM dataset WHERE name='dnssec_qtype'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='certain_qnames_vs_qtype'), (SELECT id FROM dataset WHERE name='certain_qnames_vs_qtype'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='qtype_vs_tld'), (SELECT id FROM dataset WHERE name='qtype_vs_tld'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='qtype_vs_qnamelen'), (SELECT id FROM dataset WHERE name='qtype_vs_qnamelen'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode'), (SELECT id FROM dataset WHERE name='rcode'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode_vs_replylen'), (SELECT id FROM dataset WHERE name='rcode_vs_replylen'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode_vs_replylen_big'), (SELECT id FROM dataset WHERE name='rcode_vs_replylen'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='dns_ip_version'), (SELECT id FROM dataset WHERE name='dns_ip_version'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='dns_ip_version_vs_qtype'), (SELECT id FROM dataset WHERE name='dns_ip_version_vs_qtype'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='direction_vs_ipproto'), (SELECT id FROM dataset WHERE name='direction_vs_ipproto'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='transport_vs_qtype'), (SELECT id FROM dataset WHERE name='transport_vs_qtype'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='pcap_stats'), (SELECT id FROM dataset WHERE name='pcap_stats'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet_accum'), (SELECT id FROM dataset WHERE name='client_subnet_accum'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_addr_vs_rcode_accum'), (SELECT id FROM dataset WHERE name='client_addr_vs_rcode_accum'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='ipv6_rsn_abusers_accum'), (SELECT id FROM dataset WHERE name='ipv6_rsn_abusers_accum'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet2_accum'), (SELECT id FROM dataset WHERE name='client_subnet2_accum'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet2_count'), (SELECT id FROM dataset WHERE name='client_subnet2_count'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='client_subnet2_trace'), (SELECT id FROM dataset WHERE name='client_subnet2_trace'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='rcode_volume'), (SELECT id FROM dataset WHERE name='rcode'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_small'), (SELECT id FROM dataset WHERE name='traffic_sizes_queries'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_small'), (SELECT id FROM dataset WHERE name='traffic_sizes_responses'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_big'), (SELECT id FROM dataset WHERE name='traffic_sizes_queries'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_sizes_big'), (SELECT id FROM dataset WHERE name='traffic_sizes_responses'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_volume'), (SELECT id FROM dataset WHERE name='traffic_volume_queries'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='traffic_volume'), (SELECT id FROM dataset WHERE name='traffic_volume_responses'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id from visible_plot where name='unique_sources'), (SELECT id FROM dataset WHERE name='unique_sources'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id FROM visible_plot WHERE name='traffic_volume_difference'), (SELECT id FROM dataset WHERE name='traffic_volume_queries'));
INSERT INTO dataset_visible_plot (visible_plot_id, dataset_id) VALUES ((SELECT id FROM visible_plot WHERE name='traffic_volume_difference'), (SELECT id FROM dataset WHERE name='traffic_volume_responses'));


INSERT INTO query_classification (name, title) VALUES ('Malformed', 'The DNS message was malformed and could not be entirely parsed');
INSERT INTO query_classification (name, title) VALUES ('Src port 0', 'The UDP query came from source port 0');
INSERT INTO query_classification (name, title) VALUES ('Funny Qtype', 'Query type was not one of the documented types');
INSERT INTO query_classification (name, title) VALUES ('Funny Qclass', 'Query class was not IN');
INSERT INTO query_classification (name, title) VALUES ('RFC1918PTR', 'The query type was PTR and the name was in an in-addr.arpa zone covered by RFC1918 private address space');
INSERT INTO query_classification (name, title) VALUES ('A-for-.', 'The query name was empty (equal to the root zone)');
INSERT INTO query_classification (name, title) VALUES ('A-for-A', 'The query name was already an IPv4 address');
INSERT INTO query_classification (name, title) VALUES ('localhost', 'The query was for localhost');
INSERT INTO query_classification (name, title) VALUES ('root-servers.net', 'The query was for a root-servers.net name');
INSERT INTO query_classification (name, title) VALUES ('Non-Authoritative TLD', ' The query was for a known-invalid TLD');
INSERT INTO query_classification (name, title) VALUES ('Unclassified', 'the query did not fall INTO one of the other categories');



INSERT INTO dsc.iana_lookup (registry,value,name,description) values
    ('IP', 1, 'ICMP', 'Internet Control Message'),
    ('IP', 6, 'TCP', 'Transmission Control'),
    ('IP', 17, 'UDP', 'User Datagram'),
    ('IP', 58, 'IPv6-ICMP', 'ICMP for IPv6');

INSERT INTO dsc.iana_lookup (registry,value,name,description) values
    ('DNS CLASS', 0, 'Reserved', 'Reserved'),
    ('DNS CLASS', 1, 'IN', 'Internet'),
    ('DNS CLASS', 2, 'Unassigned', 'Unassigned'),
    ('DNS CLASS', 3, 'CH', 'Chaos'),
    ('DNS CLASS', 4, 'HS', 'Hesiod'),
    ('DNS CLASS', 254, 'QCLASS NONE', 'QCLASS NONE'),
    ('DNS CLASS', 4, 'ANY', 'QCLASS *'),
    ('DNS CLASS', 65535, 'Reserved', 'Reserved');

INSERT INTO dsc.iana_lookup (registry,value,name,description) values
    ('qtype', 1, 'A', 'a host address'),
    ('qtype', 2, 'NS', 'an authoritative name server'),
    ('qtype', 3, 'MD', 'a mail destination (OBSOLETE - use MX)'),
    ('qtype', 4, 'MF', 'a mail forwarder (OBSOLETE - use MX)'),
    ('qtype', 5, 'CNAME', 'the canonical name for an alias'),
    ('qtype', 6, 'SOA', 'marks the start of a zone of authority'),
    ('qtype', 7, 'MB', 'a mailbox domain name (EXPERIMENTAL)'),
    ('qtype', 8, 'MG', 'a mail group member (EXPERIMENTAL)'),
    ('qtype', 9, 'MR', 'a mail rename domain name (EXPERIMENTAL)'),
    ('qtype', 10, 'NULL', 'a null RR (EXPERIMENTAL)'),
    ('qtype', 11, 'WKS', 'a well known service description'),
    ('qtype', 12, 'PTR', 'a domain name pointer'),
    ('qtype', 13, 'HINFO', 'host information'),
    ('qtype', 14, 'MINFO', 'mailbox or mail list information'),
    ('qtype', 15, 'MX', 'mail exchange'),
    ('qtype', 16, 'TXT', 'text strings'),
    ('qtype', 17, 'RP', 'for Responsible Person'),
    ('qtype', 18, 'AFSDB', 'for AFS Data Base location'),
    ('qtype', 19, 'X25', 'for X.25 PSDN address'),
    ('qtype', 20, 'ISDN', 'for ISDN address'),
    ('qtype', 21, 'RT', 'for Route Through'),
    ('qtype', 22, 'NSAP', 'for NSAP address, NSAP style A record'),
    ('qtype', 23, 'NSAP-PTR', 'for domain name pointer, NSAP style'),
    ('qtype', 24, 'SIG', 'for security signature'),
    ('qtype', 25, 'KEY', 'for security key'),
    ('dnssec_qtype', 25, 'KEY', 'for security key'),
    ('qtype', 26, 'PX', 'fX.400 mail mapping information'),
    ('qtype', 27, 'GPOS', 'Geographical Position'),
    ('qtype', 28, 'AAAA', 'IP6 Address'),
    ('qtype', 29, 'LOC', 'Location Information'),
    ('qtype', 30, 'NXT', 'Next Domain (OBSOLETE)'),
    ('qtype', 31, 'EID', 'Endpoint Identifier'),
    ('qtype', 32, 'NIMLOC', 'Nimrod Locator'),
    ('qtype', 33, 'SRV', 'Server Selection'),
    ('qtype', 34, 'ATMA', 'ATM Address'),
    ('qtype', 35, 'NAPTR', 'Naming Authority Pointer'),
    ('qtype', 36, 'KX', 'Key Exchanger'),
    ('qtype', 37, 'CERT', 'CERT'),
    ('qtype', 38, 'A6', 'A6 (OBSOLETE - use AAAA)'),
    ('qtype', 39, 'DNAME', 'DNAME'),
    ('qtype', 40, 'SINK', 'SINK'),
    ('qtype', 41, 'OPT', 'OPT'),
    ('qtype', 42, 'APL', 'APL'),
    ('qtype', 43, 'DS', 'Delegation Signer'),
    ('dnssec_qtype', 43, 'DS', 'Delegation Signer'),
    ('qtype', 44, 'SSHFP', 'SSH Key Fingerprint'),
    ('qtype', 45, 'IPSECKEY', 'IPSECKEY'),
    ('qtype', 46, 'RRSIG', 'RRSIG'),
    ('dnssec_qtype', 46, 'RRSIG', 'RRSIG'),
    ('qtype', 47, 'NSEC', 'NSEC'),
    ('dnssec_qtype', 47, 'NSEC', 'NSEC'),
    ('qtype', 48, 'DNSKEY', 'DNSKEY'),
    ('dnssec_qtype', 48, 'DNSKEY', 'DNSKEY'),
    ('qtype', 49, 'DHCID', 'DHCID'),
    ('qtype', 50, 'NSEC3', 'NSEC3'),
    ('dnssec_qtype', 50, 'NSEC3', 'NSEC3'),
    ('qtype', 51, 'NSEC3PARAM', 'NSEC3PARAM'),
    ('dnssec_qtype', 51, 'NSEC3PARAM', 'NSEC3PARAM'),
    ('qtype', 52, 'TLSA', 'TLSA'),
    ('qtype', 55, 'HIP', 'Host Identity Protocol'),
    ('qtype', 56, 'NINFO', 'NINFO'),
    ('qtype', 57, 'RKEY', 'RKEY'),
    ('qtype', 58, 'TALINK', 'Trust Anchor LINK'),
    ('qtype', 59, 'CDS', 'Child DS'),
    ('qtype', 99, 'SPF', NULL),
    ('qtype', 100, 'UINFO', NULL),
    ('qtype', 101, 'UID', NULL),
    ('qtype', 102, 'GID', NULL),
    ('qtype', 103, 'UNSPEC', NULL),
    ('qtype', 104, 'NID', NULL),
    ('qtype', 105, 'L32', NULL),
    ('qtype', 106, 'L64', NULL),
    ('qtype', 107, 'LP', NULL),
    ('qtype', 249, 'TKEY', 'Transaction Key'),
    ('qtype', 250, 'TSIG', 'Transaction Signature'),
    ('qtype', 251, 'IXFR', 'incremental transfer'),
    ('qtype', 252, 'AXFR', 'transfer of an entire zone'),
    ('qtype', 253, 'MAILB', 'mailbox-related RRs (MB, MG or MR)'),
    ('qtype', 254, 'MAILA', 'mail agent RRs (OBSOLETE - see MX)'),
    ('qtype', 255, 'ANY', 'A request for all records'),
    ('qtype', 256, 'URI', 'URI'),
    ('qtype', 257, 'CAA', 'Certification Authority Authorization'),
    ('qtype', 32768, 'TA', 'DNSSEC Trust Authorities'),
    ('qtype', 32769, 'DLV', 'DNSSEC Lookaside Validation'),
    ('qtype', 65535, 'Reserved', NULL);

INSERT INTO dsc.iana_lookup (registry,value,name,description) values
    ('opcode', 0, 'Query', NULL),
    ('opcode', 1, 'IQuery', '(Inverse Query, OBSOLETE)'),
    ('opcode', 2, 'Status', NULL),
    ('opcode', 3, 'Unassigned', NULL),
    ('opcode', 4, 'Notify', NULL),
    ('opcode', 5, 'Update', NULL);

INSERT INTO dsc.iana_lookup (registry,value,name,description) values
    ('rcode', 0, 'NoError', 'No Error'),
    ('rcode', 1, 'FormErr', 'Format Error'),
    ('rcode', 2, 'ServFail', 'Server Failure'),
    ('rcode', 3, 'NXDomain', 'Non-Existent Domain'),
    ('rcode', 4, 'NotImp', 'Not Implemented'),
    ('rcode', 5, 'Refused', 'Query Refused'),
    ('rcode', 6, 'YXDomain', 'Name Exists when it should not'),
    ('rcode', 7, 'YXRRSet', 'RR Set Exists when it should not'),
    ('rcode', 8, 'NXRRSet', 'RR Set that should exist does not'),
    ('rcode', 9, 'NotAuth', 'Server Not Authoritative for zone'),
    ('rcode', 10, 'NotZone', 'Name not contained in zone'),
    ('rcode', 16, 'BADVERS', 'Bad OPT Version'),
    ('rcode', 17, 'BADKEY', 'Key not recognized'),
    ('rcode', 18, 'BADTIME', 'Signature out of time window'),
    ('rcode', 19, 'BADMODE', 'Bad TKEY Mode'),
    ('rcode', 20, 'BADNAME', 'Duplicate key name'),
    ('rcode', 21, 'BADALG', 'Algorithm not supported'),
    ('rcode', 22, 'BADTRUNC', 'Bad Truncation'),
    ('rcode', 65535, 'Reserved', 'Reserved, can be allocated by Standards Action');

INSERT INTO dsc.iana_lookup (registry,value,name,description) values
    ('DNS EDNS0 Options', 0, 'Reserved', NULL),
    ('DNS EDNS0 Options', 1, 'LLQ', 'On-hold'),
    ('DNS EDNS0 Options', 2, 'UL', 'On-hold'),
    ('DNS EDNS0 Options', 3, 'NSID', 'Standard'),
    ('DNS EDNS0 Options', 4, 'Reserved', NULL);

-- Fill in the blanks
INSERT INTO dsc.iana_lookup
(
SELECT 'DNS CLASS' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(5,253) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'DNS CLASS' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(256,65279) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'DNS CLASS' AS registry, v AS value, 'Reserved for Private Use' AS name
FROM generate_series(65280,65534) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(53,54) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(60,98) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(108,248) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(258,32767) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'qtype' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(32770,65279) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'qtype' AS registry, v AS value, 'Reserved for Private Use' AS name
FROM generate_series(65280,65534) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'opcode' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(6,15) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'rcode' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(11,15) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'rcode' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(23,3840) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'rcode' AS registry, v AS value, 'Reserved for Private Use' AS name
FROM generate_series(3841,4095) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'rcode' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(4096,65534) as v
);

INSERT INTO dsc.iana_lookup
(
SELECT 'DNS EDNS0 Options' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(5,65534) as v
);

-- These are flags - not sure where I am going to put them.
/*
INSERT INTO dsc.iana_lookup (registry,value,name,description) values
    ('DNS Header Flags', 5, 'AA', 'Authoritative Answer'),
    ('DNS Header Flags', 6, 'TC', 'Truncated Response'),
    ('DNS Header Flags', 7, 'RD', 'Recursion Desired'),
    ('DNS Header Flags', 8, 'RA', 'Recursion Allowed'),
    ('DNS Header Flags', 9, 'Reserved', 'Reserved'),
    ('DNS Header Flags', 10, 'AD', 'Authentic Data'),
    ('DNS Header Flags', 11, 'CD', 'Checking Disabled');

INSERT INTO dsc.iana_lookup (registry,value,name,description) values ('EDNS Header Flags', 0, 'DO', 'DNSSEC answer OK');
INSERT INTO dsc.iana_lookup
(
SELECT 'EDNS Header Flags' AS registry, v AS value, 'Reserved' AS name
FROM generate_series(5,65534) as v
);

 INSERT INTO dsc.iana_lookup (registry,value,name,description) values ('EDNS version Number', 0, 'EDNS version 0', 'EDNS version 0');
INSERT INTO dsc.iana_lookup
(
SELECT 'EDNS version Number' AS registry, v AS value, 'Unassigned' AS name
FROM generate_series(5,65534) as v
);
*/
#!@PYTHON_EXE@
# 
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#
# Developed by Sinodun IT (www.sinodun.com)
#
import csv
with open('/tmp/GeoLite2-City-Blocks-IPv4.csv', 'rb') as f:
	reader = csv.reader(f)
	with open('/tmp/blocks_ready_to_import.csv', 'wb') as o:
		writer = csv.writer(o)
		for row in reader:
			if row[0] == "network":
				continue
			if row[1] == "":
				writer.writerow([row[0],row[2],row[7],row[8]])
			else:
				writer.writerow([row[0],row[1],row[7],row[8]])
	o.close()
f.close()
with open('/tmp/GeoLite2-City-Blocks-IPv6.csv', 'rb') as f:
	reader = csv.reader(f)
	with open('/tmp/blocks_ready_to_import.csv', 'ab') as o:
		writer = csv.writer(o)
		for row in reader:
			if row[0] == "network":
				continue
			if row[1] == "":
				writer.writerow([row[0],row[2],row[7],row[8]])
			else:
				writer.writerow([row[0],row[1],row[7],row[8]])
	o.close()
f.close()
#!@PYTHON_EXE@
# 
# Copyright 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#
# Developed by Sinodun IT (www.sinodun.com)
#
import csv
with open('/tmp/GeoLite2-City-Locations-en.csv', 'rb') as f:
	reader = csv.reader(f)
	with open('/tmp/locations_ready_to_import.csv', 'ab') as o:
		writer = csv.writer(o)
		for row in reader:
			if row[0] == "geoname_id":
				continue
			writer.writerow([row[0],row[5],row[10]])
	o.close()
f.close()
#!/usr/bin/env bash
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

# Wrapper script to run the other tools
# The other tools will be installed in libexecdir

set -e
# set -x 

ALL_TOOLS=( database_init \
        database_manage_partitions \
        database_process_rssac_data \
        database_rm_old_rssac_data \
        database_update_geoip \
        database_update_nodes \
        database_update_service_addr \
        database_update_tlds \
        database_update_tlds_from_zone \
        datafiles_create_summary \
        datafiles_rm_empty_xml_dirs \
        datafiles_tar_old_xml_files \
        plotcache_generate_cached_plots \
        plotcache_rm_cached_plots \
        rssac_generate_reports )

SCRIPT_DIR=@LIBEXEC@
HELP=0
FIRST_COMMAND_ARG=2
COMMAND=${1}
shopt -s nocasematch


usage () {
	echo
	echo "Run a Hedgehog command"
	echo
	echo "Usage: $(basename $0) [help|list] [COMMAND] [command_options]"
	echo
  if [ -z "$1" ] ; then
	  echo "The list of COMMANDs is:"
  	printf '  %s\n' "${TOOLS[@]}"
  else
    echo "The list of available COMMANDs can be found by using the list option."
  fi
	echo
  exit 1
}

if [[  " ${1} " == " help2man " ]] ; then
  usage help2man
fi

# Create new list from just commands that are installed
for cmd in ${ALL_TOOLS[@]} ; do
  if [ -x ${SCRIPT_DIR}/${cmd} ] ; then
    TOOLS=("${TOOLS[@]}" $cmd)
  fi
done

if [[  " ${1} " == " list " ]] ; then
  echo "Valid commands are:"
  printf '  %s\n' "${TOOLS[@]}"
  echo
  exit 0
fi

if [[  " ${1} " == " help " ]] ; then
  COMMAND=${2}
  HELP=1
  if [ -z ${COMMAND} ] ; then
    usage
  fi
fi

if [[ ! " ${TOOLS[*]} " =~ " ${COMMAND} " ]] ; then
  echo
  echo "Invalid command: ${COMMAND}"
  usage
fi

if [[ ${HELP} -eq 1 ]] ; then
  ${SCRIPT_DIR}/${COMMAND} -h
  exit 0
fi

exec ${SCRIPT_DIR}/${COMMAND} ${@:$FIRST_COMMAND_ARG} 
#!/bin/bash
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

# Developed by Sinodun IT (www.sinodun.com)

#
# File: refile_and_grok  
#

PROG=$(basename "$0" )

# We need a write user with access to the data dir
a=$(@LIBEXEC@/check_user_perms -wfc)
if [ $? -ne 0 ] ; then
	echo $a
	exit 1
fi
eval $a
export PGPASSWORD=$PGPASSWORD

set -e # exit if any command yields a non 0 exit status
WD="$DATADIR"
INPUT="XML"
OUTPUT="DB"
START_DATE=""
RSSAC_FLAG=""
RESERVED_CPUS=""
PROCESSES=""
ARGS=""
LOG="dsc-extractor.out"
APPEND_LOG=0
NUM_RUNNING=0

usage () {
    echo "${PROG} - finds all input files in working directory and processes to output format"
    echo 
    echo "-w Working directory to search for input files (default: ${WD})"
    echo
    echo "-i Input file format <XML|DAT> (default: ${INPUT})"
    echo
    echo "-o Output file format <DAT|DB> (default: ${OUTPUT})"
    echo
    echo "-c Non-interactive mode - use this flag when being run by a cron job"
    echo
    echo "-s Start date from which to process incoming data (XML input only)"
    echo
    echo "-r Disable processing of rssac data. Default is to process all data."
    echo
    echo "-R Reserved real cores. Number of CPU cores to exclude from import (default 0). Specify either -R or -P."
    echo
    echo "-P Number of processes to launch (default 1 per real CPU core). Specify either -R or -P."
    echo
    echo "-a Append output to the $PROG.stdout file (default is overwrite)"
    echo
    echo "-h Show this help."
    echo
}


while getopts ":w:i:o:cs:raR:P:h" opt; do
    case "$opt" in
        w  ) WD=$OPTARG ;;
        i  ) INPUT=$OPTARG ;;
        o  ) OUTPUT=$OPTARG ;;
        s  ) START_DATE="-s $OPTARG" ;;
        r  ) RSSAC_FLAG="-r" ;;
        R  ) RESERVED_CPUS=$OPTARG ;;
        P  ) PROCESSES=$OPTARG ;;
        c  ) CRON=1 ;;
        a  ) APPEND_LOG=1 ;;
        h  ) usage
             exit 1 ;;
        \? ) usage
             exit 1 ;;
    esac
done

if [[ -n "$RESERVED_CPUS"  &&  -n "$PROCESSES" ]] ; then
	echo "Reserved CPUs ($RESERVED_CPUS) and Number of processes ($PROCESSES) cannot both be specified. Exiting."
	exit
fi

if [[ -n "$PROCESSES"  &&  ! "$PROCESSES" =~ ^[0-9]+$ ]] ; then
	echo "Number of processes ($PROCESSES) is not a positive number. Exiting."
	exit
fi

if [[ -n "$RESERVED_CPUS"  && ! "$RESERVED_CPUS" =~ ^[0-9]+$ ]] ; then
	echo "Reserved CPUS ($RESERVED_CPUS) is not a positive number. Exiting."
	exit
fi
#
# if [[ -z "$RESERVED_CPUS" ]] ; then
#     RESERVED_CPUS=0
# fi

# Get number of real CPUs
export REAL_CPUS=$(lscpu | awk -F: 'BEGIN { IGNORECASE=1; total=1 } /socket/ { total=total * $2 } END { print total }')

# At this point we have either reserved or used or nothing set
if [[ -n "$RESERVED_CPUS" ]] ; then
    if [ "$RESERVED_CPUS" -ge "$REAL_CPUS" ] ; then
    	echo "Reserved CPUS ($RESERVED_CPUS) is greater than or equal to the number of available real CPUS ($REAL_CPUS). Exiting."
    	exit
    fi
    # This option allows some CPUS to be reserved for the web front end in the case that the import is running continuously
    CPUS=$(( REAL_CPUS - RESERVED_CPUS ))
    echo "Using $CPUS processes on $REAL_CPUS real CPUs ($RESERVED_CPUS real CPUs reserved)."
elif [[ -n "$PROCESSES" ]] ; then
    CPUS=$PROCESSES
    echo "Using $CPUS Processes as requested by user."
else
    # The user didn't specify anything so fallback to defaults
    CPUS=$REAL_CPUS
    echo "Using $CPUS processes on $CPUS real CPUs (no real CPUs reserved)."
fi


# Check to see if dsc-extractor is already running and if it is then only start
# CPUS - num_running_dsc-extractors
NUM_RUNNING=$(ps -ef | grep -v grep | awk ' { print $8 } ' | grep dsc-extractor | wc -l)
CPUS=$(( CPUS - NUM_RUNNING ))
if [ $CPUS -le 0 ] ; then
    echo "$(date) Too many dsc-extractor processes already running, exiting"
    exit 1
fi
echo "$(date) Starting $CPUS new processes."

if [[ "$INPUT" = "XML" || "$INPUT" = "DAT" ]] && [ "$OUTPUT" = "DB" ]
then
    ARGS="-i ${INPUT} -o PG_DB -c \"${CONN_STRING_XX}\" ${START_DATE} ${RSSAC_FLAG}"
    LOG=${LOG/.out/-xml-db.out}
    [ "$INPUT" == "DAT" ] && LOG=${LOG/xml/dat}
fi


if [ -z "$CRON" ]
then
    echo "Running ${PROG} ${INPUT}->${OUTPUT} from ${WD}"
    echo "ARGS = $ARGS"
    echo "LOG = $LOG"
    echo -n "Would you like to continue? y/N: "
    read c
    [[ ! "$c" =~ ^(Y|y) ]] && exit 1
fi

#set -x #Print commands just before execution - with all expansions and substitutions done, and words marked - useful for debugging.

cd "${WD}"
if [ $APPEND_LOG -eq 1 ]; then
	exec >>"$PROG.stdout"
else 
	exec >"$PROG.stdout"
fi
#exec 2>&1

echo "$(date) SCRIPT COMMENCED"
BINDIR=@BIN@
export BINDIR SERVER
for SERVER in $(psql $CONN_STRING -tc "select display_name from server;") ; do
    [ -L "$SERVER" ] && continue
    [ -d "$SERVER" ] || continue
    ls "${SERVER}" | xargs -i --max-procs=$CPUS bash -c "[ -L \"${SERVER}/{}\" ] || ([ -d \"${SERVER}/{}\" ] && cd \"${SERVER}/{}\"; echo \"\$(date) START: ${SERVER}/{}\"; $BINDIR/dsc-extractor ${ARGS} >> \"${LOG}\" 2>&1; echo \"\$(date) DONE: ${SERVER}/{}\")"
    # use below version instead to skip nodes that don't
    # have an incoming directory.  It is useful if you
    # receive pre-grokked data to this presenter, but
    # NOT useful if you need backward compatibility with
    # older collectors that do not put XMLs into an
    # incoming subdirectory
    # ls "${SERVER}" | xargs -i --max-procs=$CPUS bash -c "[ -L \"${SERVER}/{}\" ] || ([ -d \"${SERVER}/{}\" ] && [ -d \"${SERVER}/{}/incoming\" ] && cd \"${SERVER}/{}\"; echo \"`date`-->START: ${SERVER}/{}\"; $BINDIR/dsc-extractor ${ARGS} >> \"${LOG}\" 2>&1; echo \"`date`-->DONE: ${SERVER}/{}\")"
done
echo "$(date) SCRIPT COMPLETE"
# TODO: Split this makefile up
# TODO: misleading error message when trying to remove RPostgreSQLHelper if not there: sudo R CMD REMOVE RPostgreSQLHelper || true

CLEANFILES = $(data_DATA) $(nobase_data_DATA)

sysconfdir = @sysconfdir@/hedgehog

nobase_data_DATA =  hedgehog/conf/hedgehog.conf \
					hedgehog/conf/hedgehog_webdav.conf \
					hedgehog/www/R/startup.R \
					hedgehog/www/hedgehog.html \
					hedgehog/www/about.html \
					hedgehog/www/brew/hedgehog.brew 

nobase_dist_data_DATA = hedgehog/www/images/ajax-loader.gif \
						hedgehog/www/images/apple-touch-icon-114x114.png \
						hedgehog/www/images/apple-touch-icon.png \
						hedgehog/www/images/apple-touch-icon-72x72.png \
						hedgehog/www/images/calendar.png \
						hedgehog/www/images/favicon.ico \
						hedgehog/www/images/none.png \
						hedgehog/www/images/some.png \
						hedgehog/www/images/all.png \
						hedgehog/www/images/info.png \
						hedgehog/www/images/help.png \
						hedgehog/www/js/datetimes.js \
						hedgehog/www/js/gengraph.js \
						hedgehog/www/js/nodetabs.js \
						hedgehog/www/plots/no_graph.png \
						hedgehog/www/plots/no_results.png \
						hedgehog/www/plots/no_connection.png \
						hedgehog/www/R/generatePlot.R \
						hedgehog/www/R/prepareStatement.R \
						hedgehog/www/R/utilities.R \
						hedgehog/www/stylesheets/base.css \
						hedgehog/www/stylesheets/layout.css \
						hedgehog/www/stylesheets/skeleton.css \
						hedgehog/R_packages/RPostgreSQLHelper/DESCRIPTION \
						hedgehog/R_packages/RPostgreSQLHelper/man/dbGetDataFrame.Rd \
						hedgehog/R_packages/RPostgreSQLHelper/man/dbGetResultSet.Rd \
						hedgehog/R_packages/RPostgreSQLHelper/man/maxAttempts.Rd \
						hedgehog/R_packages/RPostgreSQLHelper/man/RPostgreSQLHelper-package.Rd \
						hedgehog/R_packages/RPostgreSQLHelper/NAMESPACE \
						hedgehog/R_packages/RPostgreSQLHelper/R/dbGetFunctions.r \
						hedgehog/conf/rssac.yaml
subst_PACKAGEVERSION = sed -e "s,@HH_PACKAGE_VERSION@,@PACKAGE_VERSION@," -e "s|@HH_VERSION_NAME@|@VERSION_NAME@|"
subst_DB_SCHEMA_VERSION = sed -e 's|DB_SCHEMA_VERSION|@DB_SCHEMA_VERSION@|'

hedgehog/conf/hedgehog.conf: $(srcdir)/hedgehog/conf/hedgehog.conf.in Makefile
	mkdir -p hedgehog/conf/
	$(subst_HH_DIRS) < $(srcdir)/hedgehog/conf/hedgehog.conf.in > hedgehog/conf/hedgehog.conf

hedgehog/conf/Rapache_hedgehog.load: $(srcdir)/hedgehog/conf/Rapache_hedgehog.load.in Makefile
	mkdir -p hedgehog/conf/
	$(subst_HH_DIRS) < $(srcdir)/hedgehog/conf/Rapache_hedgehog.load.in > hedgehog/conf/Rapache_hedgehog.load

hedgehog/www/R/startup.R: $(srcdir)/hedgehog/www/R/startup.R.in Makefile
	mkdir -p hedgehog/www/R/
	$(subst_HH_CONF_FILE) < $(srcdir)/hedgehog/www/R/startup.R.in > hedgehog/www/R/startup.R	

hedgehog/www/hedgehog.html: $(srcdir)/hedgehog/www/hedgehog.html.in Makefile
	$(subst_PACKAGEVERSION) < $(srcdir)/hedgehog/www/hedgehog.html.in > hedgehog/www/hedgehog.html

hedgehog/www/about.html: $(srcdir)/hedgehog/www/about.html.in Makefile
	$(subst_PACKAGEVERSION) < $(srcdir)/hedgehog/www/about.html.in > hedgehog/www/about.html

hedgehog/www/brew/hedgehog.brew: $(srcdir)/hedgehog/www/brew/hedgehog.brew.in Makefile
	mkdir -p hedgehog/www/brew/
	$(subst_DB_SCHEMA_VERSION) < $(srcdir)/hedgehog/www/brew/hedgehog.brew.in | $(subst_HH_DIRS)  > hedgehog/www/brew/hedgehog.brew

EXTRA_DIST = 	$(srcdir)/hedgehog/www/hedgehog.html.in \
				$(srcdir)/hedgehog/www/about.html.in \
				$(srcdir)/hedgehog/www/R/startup.R.in \
				$(srcdir)/hedgehog/conf/hedgehog.conf.in \
				$(srcdir)/hedgehog/conf/Rapache_hedgehog.load.in \
				$(srcdir)/hedgehog/www/brew/hedgehog.brew.in

install-data-hook:
	mkdir -p ${DESTDIR}$(localstatedir)/hedgehog/www
	chmod -R 775 ${DESTDIR}$(localstatedir)/hedgehog/www
	test -f ${DESTDIR}${sysconfdir}/hedgehog_gui.yaml || \
		${INSTALL_DATA} -m 644 $(srcdir)/$(hh_web_conf_dir)/hedgehog_gui.yaml ${DESTDIR}${sysconfdir}
		${INSTALL_DATA} -m 644 $(srcdir)/$(hh_web_conf_dir)/hedgehog_gui.yaml ${DESTDIR}${sysconfdir}/hedgehog_gui.yaml.sample
Package: RPostgreSQLHelper
Title: wrapper functions for the RPostgreSQL package to handle db
        connection failures, where connections are initialised via
        rapache config file.
Version: 1.0
Depends: RPostgreSQL
Date: 2013-03-20
Author: John Dickinson <jad@sinodun.com>
Maintainer: John Dickinson <jad@sinodun.com>
Description: Attempts to reconnect to the database if the
	connection has been lost and fails gracefully in
        the event that the connection to the database
        cannot be re-established.
License: GPL-3
Packaged: 2013-03-20 16:11:46 UTC; jad
#
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.# 
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#

#
# Developed by Sinodun IT (www.sinodun.com)
#

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# EXPORTS
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Export all public methods, that is, those without a preceeding dot
# in their names.
exportPattern("^[^\\.]")
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.
#

#
# Developed by Sinodun IT (www.sinodun.com)
#

#   wrapper functions for RPostgreSQL dbSendQuery and
#   fetch to handle db connection failures, where
#   connections are initialised via rapache config file.
#   
#   ensures that dbSendQuery first attempts to
#   reconnect to the database if the connection has been
#   lost (apache2 force-reload) and fails gracefully in
#   the event that the connection to the database cannot
#   be re-established.
#   
#   dbGetResultSet() will return a result set if successful
#   or NULL otherwise.
#   
#   dbGetDataFrame() will return a data frame if successful
#   or NULL otherwise.
#   
#   dependencies:   RPostgreSQL
#   
#   author: John Dickinson <jad@sinodun.com>


library("RPostgreSQL")

# max number of attempts to reconnect to the database
maxAttempts <- 3


#   attempt to re-establish a db connection
.reload <- function(dbiDriver, dbiConnection, dbConnStr){
    # close any pending result sets
    resSets = try(dbListResults(dbiConnection))
    if(class(resSets) != "try-error"){
        for(resultSet in resSets){
            dbClearResult(resultSet)
        }
    }

    # close all connections that the dbiDriver is managing
    conns = try(dbListConnections(dbiDriver))
    if(class(conns) != "try-error"){
        for(connection in conns){
            dbDisconnect(connection)
        }
    }

    dbiConnection <- eval(parse(text = paste("dbConnect(dbdrv,",  dbConnStr,")")))

}

#   return the result set associated with sqlQueryStr
#   from dbiConnection. If the dbiConnection has been lost,
#   attempt to reconnect to the db. If dbiConnection cannot
#   be established, return NULL
dbGetResultSet <- function(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr){
    rs <- NULL
    for(attempt in 1:(maxAttempts+1)){
        rs <- try(dbSendQuery(dbiConnection ,sqlQueryStr))
        if(class(rs) == "try-error"){
            if(attempt == 1){
                system('logger -p user.notice Hedgehog: Error sending query to database')
            }
            if(attempt == (maxAttempts+1)){
                system('logger -p user.notice Hedgehog: Critical Giving up trying to send query to the database: administrator action required')
                rs <- NULL
            }else{
                system(paste("logger -p user.notice Hedgehog: Info re-connecting to the database: attempt ", attempt, " of ", maxAttempts, sep=""))
                .reload(dbiDriver, dbiConnection, dbConnStr)
            }
        }else break
    }
    return(rs)
}

#   return the data frame associated with sqlQueryStr
#   from dbiConnection. 
dbGetDataFrame <- function(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr){
    df <- NULL
    rs <- dbGetResultSet(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
    if(!is.null(rs)){
        df <- fetch(rs, n=-1)
    dbClearResult(rs)
    }
    return(df)
}
%%
%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
%%
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, you can obtain one at https://mozilla.org/MPL/2.0/.

%%
%% Developed by Sinodun IT (www.sinodun.com)
%%

\name{RPostgreSQLHelper-package}
\alias{RPostgreSQLHelper-package}
\alias{RPostgreSQLHelper}
\docType{package}
\title{RPostgreSQL wrapper functions}
\description{Handles dbiConnection failures}
\details{
\tabular{ll}{
Package: \tab RPostgreSQLHelper\cr
Type: \tab Package\cr
Version: \tab 1.0\cr
Date: \tab 2013-03-20\cr
License: \tab GPL-3\cr
}
dbSendQuery: dbGetResultSet(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)

dbSendQuery + fetch: dbGetDataFrame(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
}
\author{John Dickinson <jad@sinodun.com>}
\references{
	https://code.google.com/p/rpostgresql/
	http://stat.bell-labs.com/RS-DBI/doc/html/index.html
}
%%~~ Optionally other standard keywords, one per line, from file KEYWORDS in ~~
%%~~ the R documentation directory ~~
\keyword{ package }
\seealso{
%%~~ Optional links to other man pages, e.g. ~~
%%~~ \code{\link[<pkg>:<pkg>-package]{<pkg>}} ~~
}
\examples{
%%~~ simple examples of the most important functions ~~
}
%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
%%
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, you can obtain one at https://mozilla.org/MPL/2.0/.

%%
%% Developed by Sinodun IT (www.sinodun.com)
%%

\name{dbGetDataFrame}
\alias{dbGetDataFrame}
%- Also NEED an '\alias' for EACH other topic documented here.
\title{RPostgreSQL dbSendQuery + fetch wrapper function}
\description{Handles dbiConnection failures}
\usage{
dbGetDataFrame(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
}
%- maybe also 'usage' for other objects documented here.
\arguments{
  \item{dbiDriver}{
Database driver dbiDriver
}
  \item{dbiConnection}{
Database connection dbiConnection
}
  \item{dbConnStr}{
Database connection string dbConnStr
}
  \item{sqlQueryStr}{
String containing SQL query to be run
}
}
\details{
%%  ~~ If necessary, more details than the description above ~~
}
\value{if successful, return a DBIResult else returns NULL}
\references{
	https://code.google.com/p/rpostgresql/
	http://stat.bell-labs.com/RS-DBI/doc/html/index.html
}
\author{John Dickinson <jad@sinodun.com>}
\note{
%%  ~~further notes~~
}

%% ~Make other sections like Warning with \section{Warning }{....} ~

\seealso{
%% ~~objects to See Also as \code{\link{help}}, ~~~
}
\examples{
}
% Add one or more standard keywords, see file 'KEYWORDS' in the
% R documentation directory.
\keyword{ ~kwd1 }
\keyword{ ~kwd2 }% __ONLY ONE__ keyword per line
%%
%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
%%
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, you can obtain one at https://mozilla.org/MPL/2.0/.
%%
%% Developed by Sinodun IT (www.sinodun.com)
%%

\name{dbGetResultSet}
\alias{dbGetResultSet}
%- Also NEED an '\alias' for EACH other topic documented here.
\title{RPostgreSQL dbSendQuery wrapper function}
\description{Handles dbiConnection failures}
\usage{
dbGetResultSet(dbiDriver, dbiConnection, dbConnStr, sqlQueryStr)
}
%- maybe also 'usage' for other objects documented here.
\arguments{
  \item{dbiDriver}{
Database driver dbiDriver
}
  \item{dbiConnection}{
Database connection dbiConnection
}
  \item{dbConnStr}{
Database connection string dbConnStr
}
  \item{sqlQueryStr}{
String containing SQL query to be run
}
}
\details{
%%  ~~ If necessary, more details than the description above ~~
}
\value{if successful, return a DBIResult else returns NULL}
\references{
	https://code.google.com/p/rpostgresql/
	http://stat.bell-labs.com/RS-DBI/doc/html/index.html
}
\author{John Dickinson <jad@sinodun.com>}
\note{
%%  ~~further notes~~
}

%% ~Make other sections like Warning with \section{Warning }{....} ~

\seealso{
%% ~~objects to See Also as \code{\link{help}}, ~~~
}
\examples{
}
% Add one or more standard keywords, see file 'KEYWORDS' in the
% R documentation directory.
\keyword{ ~kwd1 }
\keyword{ ~kwd2 }% __ONLY ONE__ keyword per line
%%
%% Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
%%
%% This Source Code Form is subject to the terms of the Mozilla Public
%% License, v. 2.0. If a copy of the MPL was not distributed with this
%% file, you can obtain one at https://mozilla.org/MPL/2.0/.

%%
%% Developed by Sinodun IT (www.sinodun.com)
%%

\name{maxAttempts}
\alias{maxAttempts}
\docType{data}
\title{maximum number of attempts to re-connect to the database}
\description{integer value indicating the maximum number of attempts to re-connect to the database}
\usage{}
\format{
  The format is:
 num 3
}
\details{
%%  ~~ If necessary, more details than the __description__ above ~~
}
\source{
%%  ~~ reference to a publication or URL from which the data were obtained ~~
}
\references{
%%  ~~ possibly secondary sources and usages ~~
}
\examples{
data(maxAttempts)
}
\keyword{datasets}
# 
## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
##
## This Source Code Form is subject to the terms of the Mozilla Public
## License, v. 2.0. If a copy of the MPL was not distributed with this
## file, you can obtain one at https://mozilla.org/MPL/2.0/.

## Developed by Sinodun IT (www.sinodun.com)

<VirtualHost *:80>

	#ServerName www.example.com
	ServerAdmin webmaster@localhost
  
	Alias /hedgehog @HH_WEB_SOURCE@
	<Directory @HH_WEB_SOURCE@>
	    RSourceOnStartup @HH_WEB_SOURCE@/R/startup.R
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    DefaultType text/html
	    Require all granted
	    DirectoryIndex hedgehog.html
	</Directory>

	<Directory @HH_WEB_SOURCE@/brew>
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    SetHandler r-script
	    Require all granted
	    RHandler brew::brew
	</Directory>

	<Directory @HH_WEB_SOURCE@/images>
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    Require all granted
	    SetHandler default-handler
	</Directory>

	<Directory @HH_WEB_SOURCE@/plots>
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    Require all granted
	    SetHandler default-handler
	</Directory>

	<Directory @HH_WEB_SOURCE@/js>
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    Require all granted
	    SetHandler default-handler
	</Directory>

	<Directory @HH_WEB_SOURCE@/R>
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    Require all granted
	    SetHandler r-script
	</Directory>

	<Directory @HH_WEB_SOURCE@/stylesheets>
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    Require all granted
	    SetHandler default-handler
	</Directory>
	
	Alias /plotcache @HH_PLOTCACHE@
	<Directory @HH_PLOTCACHE@>
	    Options None
	    AllowOverride None
	    order allow,deny
	    allow from all
	    Require all granted
	    SetHandler default-handler
	</Directory>

	Alias /rssac @HH_RSSAC@
	<Directory @HH_RSSAC@>
	    Options Indexes
	    AllowOverride None
	    order allow,deny
	    Require all granted
	    allow from all
	</Directory>	

	# Prints out a nice report about R running within Apache
	# Do not use this in production by default
	#<Location /RApacheInfo>
	#    SetHandler r-info
	#</Location>
</VirtualHost>

## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
##
## This Source Code Form is subject to the terms of the Mozilla Public
## License, v. 2.0. If a copy of the MPL was not distributed with this
## file, you can obtain one at https://mozilla.org/MPL/2.0/.

## Developed by Sinodun IT (www.sinodun.com)


--- 
# YAML config for hedgehog GUI.
# NOTE: If this file is changed then apache must be restarted for the changes to take effect
www:
  default_plot_type             : interactive # 'static'      -> png plots
                                              # 'interactive' -> googlevis plots
  default_interactive_plot_type : svg         # 'flash' -> plot requires flash
                                              # 'svg'   -> plot is SVG/VML and does not require flash
                                              # (but with svg plots some legends do not wrap properly)
  default_node_grouping         : instance    # choose from 'none', 'instance', 'city' or 'country'
  use_plot_caching              : 1           # '1' -> true, use cached plots when possible
                                              # '0' -> false, never use cached plots
  caching_delay_in_hours        : 1           # If 'use_plot_caching=1' then only plots with an end time
                                              # earlier than this number of hours ago are cached.
                                              # More recent plots are not cached as data may still be
                                              # being imported
  presentation_delay_in_hours   : 0           # Number of hours behind now for which the GUI will display data
  support_url                   :             # configurable target for "Support" external link on Homepage.
  default_server                :             # Optionally specify the default server to use in the server
                                              # drop-down (default is first alphabetically)
  default_plot_name             : by_region   # default plot name (See Hedgehog internal plot name in the user guide)
  enable_node_selection         : 1           # '1' -> true, Show the node radio button
                                              # '0' -> false, hide the node radio button
...
# <IfModule mod_ssl.c>
# <VirtualHost *:443>
#         ServerName example.com
#         SSLEngine on
#         SSLCertificateFile    Server.pem
#         SSLCertificateKeyFile Server.key
#         SSLCACertificateFile cacert.pem
#         SSLVerifyClient require
#         SSLVerifyDepth  1
#         CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined
#         DavLockDB "/somewhere/DAVLock"
#         Alias /data /var/lib/hedgehog/data
#         <Directory /var/lib/hedgehog/data>
#            DAV On
#            SSLRequireSSL
#            Options +Indexes
#            AllowOverride None
#            order allow,deny
#            allow from all
#            DirectoryIndex disable
#            Require all granted
#         </Directory>
# </VirtualHost>
# </IfModule>

## Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
##
## This Source Code Form is subject to the terms of the Mozilla Public
## License, v. 2.0. If a copy of the MPL was not distributed with this
## file, you can obtain one at https://mozilla.org/MPL/2.0/.

## Developed by Sinodun IT (www.sinodun.com)


--- # YAML config for rssac.
metric:
  traffic-volume:
    statistics_type: kv_pair
    measurement_parameter: Number of queries
    service: .root-servers.net
    statistics:
      - dns-udp-queries-received-ipv4
      - dns-udp-queries-received-ipv6
      - dns-tcp-queries-received-ipv4
      - dns-tcp-queries-received-ipv6
      - dns-udp-responses-sent-ipv4
      - dns-udp-responses-sent-ipv6
      - dns-tcp-responses-sent-ipv4
      - dns-tcp-responses-sent-ipv6
  traffic-sizes:
    statistics_type: buckets
    measurement_parameter: Query and response size distribution
    service: .root-servers.net
    statistics:
      - name: udp-request-sizes
        bucket_min: 16
        bucket_max: 288
        bucket_step: 16
      
      - name: udp-response-sizes
        bucket_min: 16
        bucket_max: 4096
        bucket_step: 16
        
      - name: tcp-request-sizes
        bucket_min: 16
        bucket_max: 288
        bucket_step: 16
        
      - name: tcp-response-sizes
        bucket_min: 16
        bucket_max: 4096
        bucket_step: 16
  rcode-volume:
    statistics_type: count
    measurement_parameter: RCODE distribution
    service: .root-servers.net
    statistics:
      - name: rcodes
  unique-sources:
    statistics_type: kv_pair_with_alias
    measurement_parameter: Number of sources seen
    service: .root-servers.net
    statistics:
      - name: num-sources-ipv4
        alias: IPv4
        
      - name: num-sources-ipv6
        alias: IPv6
        
      - name: num-sources-ipv6-aggregate
        alias: IPv6/64
  load-time:
    statistics_type: average
    measurement_parameter: Latency in publishing available data
    service: .root-servers.net
    statistics:
      - name: time
  zone-size:
    statistics_type: value
    measurement_parameter: The size of the overall root zone
    service: .root-servers.net
    statistics:
      - name: size
...
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# TODO(refactor): Refactor this code

# Define globals

# Colours
GREY <- "#999999"
ORANGE <- "#E69F00"
SKYBLUE <- "#56B4E9"
GREEN <- "#009E73"
YELLOW <- "#F0E442"
BLUE <- "#0072B2"
VERMILLION <- "#D55E00"
PINK <- "#CC79A7"
BLACK <- "#000000"
GRIDGREY <- "#BFBFBF"
DARKERRED <- "#990F0F"
DARKERORANGEBT <- "#99700F"
DARKERLIMEGREEN <- "#1F990F"
DARKERVIOLET <- "#710F99"
SOFTRED <- "#E67E7E"
SOFTORANGE <- "#E6C77E"
SOFTLIMEGREEN <- "#8AE67E"
SOFTVIOLET <- "#C77EE6"
MODERATERED <- "#CC5252"
MODERATEORANGE <- "#CCA852"
MODERATELIMEGREEN <- "#60CC52"
MODERATEVIOLET <- "#A852CC"
DARKRED <- "#B22D2D"
DARKORANGEBT <- "#B28B2D"
DARKLIMEGREEN <- "#3CB22D"
DARKVIOLET <- "#8B2DB2"

# A colour-blind friendly palette:
 CBPALETTE = c(DARKERRED, DARKERORANGEBT, DARKERLIMEGREEN, DARKERVIOLET,
               SOFTRED, SOFTORANGE, SOFTLIMEGREEN, SOFTVIOLET,
               MODERATERED, MODERATEORANGE, MODERATELIMEGREEN, MODERATEVIOLET,
               DARKRED, DARKORANGEBT, DARKLIMEGREEN, DARKVIOLET)

NCBPALETTE <- length(CBPALETTE)

MINICBPALETTE <- c(DARKERRED, DARKERLIMEGREEN)
NMINICBPALETTE <- length(MINICBPALETTE)

# width and height of ggplot pngs
W <- 940
H <- 600

##################

linePlot <- function(df, f, title, xlabel, ylabel, gvis) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In linePlot')
  }
  
    if(gvis == 1){
      # Stupid bug in gvis where the name of the numvar column causes issues with the correct name being diplayed in the legend
      names(df)[names(df)=="y"] <- "yyyyyyyyyyyy"

      if (gui_config$www$default_interactive_plot_type == "svg") {
          p <- gvisAnnotationChart(df, numvar="yyyyyyyyyyyy", idvar = "key", datevar = "x", #titlevar="serial",
                                   options=list(
                                                #displayAnnotations=TRUE, annotationsWidth=10, 
                                                legendPosition='newRow', height=540, width=910, colors="['#0072B2','#990F0F','#99700F',
      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#990F0F','#99700F','#1F990F',
      '#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99',
      '#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99','#E67E7E',
      '#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F',
      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2']"))
         
      }else{
          p <- gvisAnnotatedTimeLine(df, numvar="yyyyyyyyyyyy", idvar = "key", datevar = "x", 
                                     options=list(legendPosition='newRow', height=540, width=910, colors="['#0072B2','#990F0F','#99700F',
      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#990F0F','#99700F','#1F990F',
      '#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99',
      '#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F','#1F990F','#710F99','#E67E7E',
      '#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#0072B2','#B22D2D','#B28B2D','#3CB22D','#8B2DB2','#990F0F','#99700F',
      '#1F990F','#710F99','#E67E7E','#E6C77E','#8AE67E','#C77EE6','#CC5252','#CCA852','#60CC52','#A852CC','#B22D2D','#B28B2D','#3CB22D','#8B2DB2']"))         
      }
      title <- sub("\n", "<br />", title)
      ylabel <- gsub(" +", "&nbsp;", ylabel)
      p$html$chart['divChart'] <- paste("<div style=\"text-align: center; width: 918px; font-family: HelveticaNeue, 
                                                      'Helvetica Neue', Helvetica, Arial, sans-serif;\">", 
                                                       title,
                                        "</div>",
                                        "<div style=\"display: flex; display: -webkit-flex; justify-content:flex-start; width: 930px;\">",  
                                        "<div>", p$html$chart['divChart'], "</div>",
                                        "<div style=\"width: 17px; height: 17px; font-size: small; transform: translateY(200px) rotate(90deg); transform-origin: right top; 
                                              -webkit-transform: translateY(200px) rotate(90deg); -webkit-transform-origin: right top;\">", ylabel, "</div>",
                                        "</div>",
                                        "<div style=\"text-align: center; width: 918px; font-size: small;\">", xlabel, "</div>",
                                       sep="")
      cat(p$html$chart, file=f)
    }else{
      nKeys <- length(unique(df$key))
      df$x <- as.POSIXct(df$x)
      png(f, width = W, height = H)
      p <- ggplot(data=df, aes(x=x, y=y, group=key, colour=key)) +
                  geom_line() +
                  labs(title=title, x=xlabel, y=ylabel) +
                  scale_x_datetime(expand=c(0.01,0)) +
                  scale_y_continuous(expand=c(0.01,0), labels = comma) +
                  theme_bw() +
                  theme(panel.grid.major = element_line(colour = GRIDGREY), 
                        panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
                  guides(col = guide_legend(nrow = 20, byrow = TRUE, override.aes = list(size=3)))
      if (nKeys <= NCBPALETTE) {
        p <- p + scale_colour_manual(values=CBPALETTE)
      }

      print(p)
      dev.off()

      fstack <- sub(".png", "-stack.png", f)
      png(fstack, width = W, height = H)
      ps <- ggplot(data=df, aes(x=x, y=y, fill=key, order=key)) + 
                   geom_area(stat="identity") +
                   labs(title=title, x=xlabel, y=ylabel) +
                   scale_x_datetime(expand=c(0.01,0)) + 
                   scale_y_continuous(expand=c(0,0), labels = comma) +
                   theme_bw() +
                   theme(panel.grid.major = element_line(colour = GRIDGREY), 
                         panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
                   guides(fill = guide_legend(nrow = 20, byrow = TRUE))

      if (nKeys <= NCBPALETTE) {
        ps <- ps + scale_fill_manual(values=CBPALETTE)
      }
      print(ps)
      dev.off()

      flog <- sub(".png", "-log.png", f)
      png(flog, width = W, height = H)
      p <- p + scale_y_log10() + ylab(paste("log(",ylabel,")",sep=""))
      print(p)
      dev.off()
    }
}

barPlot <- function(df, f, title, xlabel, ylabel, gvis, vertical=0, small=1) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In barPlot')
  }

    if(gvis == 1){
      title <- sub("\n", " ", title)
      if(vertical == 1){
          p <- gvisColumnChart(df, xvar='x', yvar='y', 
                            options=list(legend="none", title=title, height=600, width=940,
                                         vAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'10'}}", sep=""), 
                                         hAxis=paste("{title:'",xlabel,"',textStyle:{fontSize:'14'}}", sep=""), 
                                         chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
      }else{
        if(small==1){
          p <- gvisBarChart(df, xvar='x', yvar='y',
                            options=list(legend="none", title=title, height=600, width=940,
                                         vAxis=paste("{title:'",xlabel,"',textStyle:{fontSize:'10'}}", sep=""), 
                                         hAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'14'}}", sep=""), 
                                         chartArea="{left:100,top:50,width:\"88%\",height:\"80%\"}"))
        } else {
          p <- gvisBarChart(df, xvar='x', yvar='y',
                            options=list(legend="none", title=title, height=600, width=940,
                                         vAxis=paste("{title:'",xlabel,"'}", sep=""), 
                                         hAxis=paste("{title:'",ylabel,"', textStyle:{fontSize:'14'}}", sep=""), 
                                         chartArea="{left:80,top:50,width:\"88%\",height:\"80%\"}"))
        }
      }
      cat(p$html$chart,file=f)
    }else{
      if (vertical == 0) {
          df$x <- factor(df$x, levels=rev(as.character(df$x)))
      }
      png(f, width = W, height = H)

      p <- ggplot(data=df, aes(x=x, y=y)) +
                  geom_bar(fill=GREY, colour=GREY, stat="identity") +
                  labs(title=title, x=xlabel, y=ylabel) +
                  theme_bw() + scale_y_continuous(expand=c(0,0), labels = comma) +
                  theme(panel.grid.major.y = element_line(colour = GRIDGREY), 
                        panel.grid.minor.y = element_line(colour = GRIDGREY, linetype = "dotted"), 
                        panel.grid.major.x = element_blank(), panel.grid.minor.x = element_blank())

      if (vertical == 0) {
        p <- p +
             coord_flip() +
             theme(panel.grid.major.x = element_line(colour = GRIDGREY), 
                   panel.grid.minor.x = element_line(colour = GRIDGREY, linetype = "dotted"), 
                   panel.grid.major.y = element_blank(), panel.grid.minor.y = element_blank())
      }

      print(p)
      dev.off()
    }
}

stackedBarPlot <- function(df, f, title, xlabel, ylabel, gvis, pltnm, scalex="discrete", vertical=0, gbar_width=0) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In stackedBarPlot')
  }
    if(gvis == 1){
      title <- sub("\n", " ", title)
      de <- cast(df, x ~ key, value='y', fun.aggregate=sum)
      y_var <- tail(colnames(de),-1)
      if(vertical == 1){
          if (gbar_width == 0) {
              p <- gvisColumnChart(de, xvar='x', yvar=y_var,
                                   options=list(isStacked=TRUE, title=title, height=600, width=940,
                                                vAxis=paste("{title:'", ylabel,"', format: 'short'}", sep=""), 
                                                hAxis=paste("{title:'", xlabel,"', textStyle:{fontSize:'14'}}", sep=""), 
                                                legend=paste("{textStyle:{fontSize:'12'}}", sep=""),
                                                chartArea="{left:80,top:50,width:\"75%\",height:\"80%\"}"))
          } else {
              p <- gvisColumnChart(de, xvar='x', yvar=y_var,
                                   options=list(isStacked=TRUE, title=title, height=600, width=940,
                                                vAxis=paste("{title:'", ylabel,"', format: 'short'}", sep=""), 
                                                hAxis=paste("{title:'", xlabel,"', textStyle:{fontSize:'14'}}", sep=""), 
                                                legend=paste("{textStyle:{fontSize:'12'}}", sep=""), 
                                                chartArea="{left:80,top:50,width:\"75%\",height:\"80%\"}", 
                                                bar=paste("{groupWidth:", gbar_width,"}", sep="")))
          }
      }else{
          # Need to order the table before passing it to gviz
          # Dont include the IP ddress in the sum!
          if (ncol(de) > 2) { 
            de$total <- rowSums(de[,-1], na.rm=TRUE);
            de <-arrange(de, desc(total));
            de$total <- NULL
          } else {
            de <-arrange(de, desc(de[[2]]));
          }
          # p <- gvisTable(de)
          p <- gvisBarChart(de, xvar='x', yvar=y_var,
                            options=list(isStacked=TRUE, title=title, height=600, width=940,
                                         vAxis=paste("{title:'",xlabel,"',textStyle:{fontSize:'10'}}", sep=""),
                                         hAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'14'}}", sep=""),
                                         legend=paste("{textStyle:{fontSize:'12'}}", sep=""),
                                         chartArea="{left:110,top:50,width:\"75%\",height:\"80%\"}"))
      }
      cat(p$html$chart,file=f)
    }else{
      if (scalex == "discrete") {
        df$x <- factor(df$x, levels=rev(as.character(df$x)))
        dfmelt <- melt(df)
        de <- cast(dfmelt, x ~ key, value='value')
        if (ncol(de) > 2) {
          de$total <- rowSums(de, na.rm=TRUE)
          dorder2 <- de[with(de, order(total)), ]
          x_order <- dorder2$x
          df$x <- factor(df$x, levels=x_order)
        }
      }

      nKeys <- length(unique(df$key))
      png(f, width = W, height = H)

      p <- ggplot(data=df, aes(x=x, y=y, fill=key, order=key)) + 
                  geom_bar(stat="identity") + 
                  labs(title=title, x=xlabel, y=ylabel) + 
                  scale_y_continuous(expand=c(0,0), labels = comma) +
                  theme_bw() +
                  theme(panel.grid.major = element_line(colour = GRIDGREY), 
                        panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
                  guides(fill = guide_legend(nrow = 20, byrow = TRUE))

      if (nKeys <= NCBPALETTE) {
        p <- p + scale_fill_manual(values=CBPALETTE)
      }

      if (scalex == "continuous") {
        p <- p +
             scale_x_continuous(expand=c(0.01,0), labels = comma)
      }

      if (vertical == 0) {
        p <- p +
             coord_flip() +
             theme(panel.grid.major.x = element_line(colour = GRIDGREY), 
                   panel.grid.minor.x = element_line(colour = GRIDGREY, linetype = "dotted"), 
                   panel.grid.major.y = element_blank(), 
                   panel.grid.minor.y = element_blank(),
                   axis.text.y=element_text(size=14))
      } else {
          p <- p +
               geom_bar(width=1, stat="identity")
      }

      if (pltnm == 'rcode_vs_replylen') {
          p <- p + 
               scale_x_continuous(expand=c(0.01,0), labels = comma, limits=c(0,1000))
      }

      if (pltnm == 'rcode_vs_replylen_big') {
          xmin <- 1000
          p <- p + 
               scale_x_continuous(expand=c(0.01,0), labels = comma, limits=c(xmin,max(df$x) + 100))
      }

      print(p)
      dev.off()
    }
}

stackedAreaPlot <- function(df, f, title, xlabel, ylabel, gvis) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In stackedAreaPlot')
  }
  
    if(gvis == 1){
      title <- sub("\n", " ", title)
      de <- cast(df, x ~ key, value='y', fun.aggregate=sum)
      y_var <- tail(colnames(de),-1)
      p <- gvisAreaChart(de, xvar='x', yvar=y_var, 
                         options=list(isStacked=TRUE, title=title, height=600, width=940, 
                                      vAxis=paste("{title:'",ylabel,"'}", sep=""), 
                                      hAxis=paste("{title:'",xlabel,"'}", sep=""), 
                                      legend=paste("{textStyle:{fontSize:'12'}}"), 
                                      chartArea="{left:80,top:50,width:\"75%\",height:\"80%\"}"))
      cat(p$html$chart,file=f)
    }else{
        df$x <- as.POSIXct(df$x)
      nKeys <- length(unique(df$key))
      png(f, width = W, height = H)

      p <- ggplot(data=df, aes(x=x, y=y, fill=key, order=key)) + 
                  geom_area(stat="identity") + 
                  labs(title=title, x=xlabel, y=ylabel) +
                  scale_x_datetime(expand=c(0.01,0)) +
                  scale_y_continuous(expand=c(0,0), labels = comma) + 
                  theme_bw() +
                  theme(panel.grid.major = element_line(colour = GRIDGREY), 
                        panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
                  guides(fill = guide_legend(nrow = 20, byrow = TRUE))

      if (nKeys <= NCBPALETTE) {
        p <- p + scale_fill_manual(values=CBPALETTE)
      }

      print(p)
      dev.off()
    }
}

facetedBarPlot <- function(df, f, title, xlabel, ylabel, gvis, bar_width) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In facetedBarPlot')
  }
  
    if(gvis == 1){
        stackedBarPlot(df, f, title, xlabel, ylabel, gvis, pltnm = 'N/A', vertical = 1, gbar_width = bar_width)
    }else{
        nKeys <- length(unique(df$key))
        png(f, width = W, height = H)

        p <- ggplot(data=df, aes(x=x, y=y)) + 
                    geom_bar(stat="identity", width=bar_width, fill=DARKERRED) +
                    facet_grid(key ~ ., scales="free") + 
                    labs(title=title, x=xlabel, y=ylabel) + 
                    theme_bw() +
                    theme(panel.grid.major = element_line(colour = GRIDGREY), 
                          panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
                    guides(fill = guide_legend(nrow = 20, byrow = TRUE))

        if (nKeys <= NCBPALETTE) {
          p <- p + scale_fill_manual(values=CBPALETTE)
        }

        print(p)
        dev.off()
    }
}

facetedLinePlot <- function(df, f, title, xlabel, ylabel, gvis) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In facetedLinePlot')
  }

    if(gvis == 1){
      # fix up the legends to work with svg
      rkey_svg <- sub("dns-(.*-.*-).*-(.*)", "\\1\\2", df$key)
      rkey_svg <- sub("responses", "resp", rkey_svg)
      df["key"] <- rkey_svg
        linePlot(df, f, title, xlabel, ylabel, gvis)
    }else{
        nKeys <- length(unique(df$key))
        png(f, width = W, height = H)
        df$x <- as.POSIXct(df$x)
        rkey1 <- sub("dns-(.*)-.*-.*-(.*)", "\\1-\\2", df$key)
        df["rkey"] <- rkey1
        rkey2 <- sub("dns-.*-(.*)-.*-.*", "\\1", df$key)
        df["key"] <- rkey2

        p <- ggplot(data=df, aes(x=x, y=y, group=key, colour=key)) +
                    geom_jitter(position=position_jitter(width=2)) +
                    geom_line() +
                    labs(title=title, x=xlabel, y=ylabel) +
                    facet_grid(rkey ~ ., scales="free") +
                    scale_x_datetime(expand=c(0.01,0)) +
                    scale_y_continuous(expand=c(0.01,0), labels=comma) +
                    theme_bw() +
                    theme(panel.grid.major = element_line(colour = GRIDGREY), 
                          panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
                    guides(col = guide_legend(nrow = 20, byrow = TRUE, override.aes=list(size=3)))

        nKeys = length(unique(df$key))
        if (nKeys <= NMINICBPALETTE) {
            p <- p + scale_colour_manual(values=MINICBPALETTE)
        }
        else if (nKeys <= NCBPALETTE) {
          p <- p + scale_colour_manual(values=CBPALETTE)
        }

        print(p)
        dev.off()
    }
}

facetedLinePlot2 <- function(df, f, title, xlabel, ylabel, gvis) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In facetedLinePlot2')
  }

    if(gvis == 1){
        linePlot(df, f, title, xlabel, ylabel, gvis)
    }else{
        nKeys <- length(unique(df$key))
        png(f, width = W, height = H)
        df$x <- as.POSIXct(df$x)
        df$facet1 <- sub(".*:tcp", "tcp", df$key)
        df$facet2 <- sub(".*:udp", "udp", df$facet1)
		write.table(df, file="/tmp/df.out")

        p <- ggplot(data=df, aes(x=x, y=y, group=key, colour=key)) +
                    geom_jitter(position=position_jitter(width=2)) +
                    geom_line() +
                    labs(title=title, x=xlabel, y=ylabel) +
                    facet_grid(facet2 ~ ., scales="free") +
                    scale_x_datetime(expand=c(0.01,0)) +
                    scale_y_continuous(expand=c(0.01,0), labels=comma) +
                    theme_bw() +
                    theme(panel.grid.major = element_line(colour = GRIDGREY), 
                          panel.grid.minor = element_line(colour = GRIDGREY, linetype = "dotted")) +
                    guides(col = guide_legend(nrow = 20, byrow = TRUE, override.aes=list(size=3)))

        nKeys = length(unique(df$key))
        if (nKeys <= NMINICBPALETTE) {
            p <- p + scale_colour_manual(values=MINICBPALETTE)
        }
        else if (nKeys <= NCBPALETTE) {
          p <- p + scale_colour_manual(values=CBPALETTE)
        }

        print(p)
        dev.off()
    }
}

facetedDiffLinePlot <- function(df, f, title, xlabel, ylabel, gvis) {

    if (hh_debug) {
      system('logger -p user.notice Hedgehog: In facetedDiffLinePlot')
    }

    nKeys <- length(unique(df$key))
    df$x <- as.POSIXct(df$x)
    rkey1 <- sub("dns-(.*)-.*-.*-(.*)", "\\1-\\2", df$key)
    png(f, width=W, height=H)
    df["rkey"] = rkey1

    dfx <- arrange(df, x, rkey, desc(key))
    dfx$y <- ifelse(test = (grepl("responses", dfx$key)), yes = -dfx$y, no = dfx$y)
    df1 <- aggregate(dfx$y, by=list(x2=dfx$x, rkey=dfx$rkey), FUN=sum)
    df1 <- plyr::rename(df1,c("x"="y", "x2"="x", "rkey"="key"))
    if (gvis == 1) {
        linePlot(df1, f, title, xlabel, ylabel, gvis)
    } else {

        p <- ggplot(data=df1, aes(x=x, y=y, group=key)) +
                    geom_line(colour=DARKERRED) +
                    labs(title=title, x=xlabel, y="Difference Between Number of Queries and Responses/min") +
                    facet_grid(key ~ ., scales="free") +
                    scale_x_datetime(expand=c(0.01,0)) +
                    scale_y_continuous(expand=c(0.01,0), labels=comma) +
                    theme_bw() +
                    theme(panel.margin=grid::unit(3,"mm"), 
                          panel.grid.major=element_line(colour=GRIDGREY),
                          panel.grid.minor=element_line(colour=GRIDGREY, linetype="dotted")) +
                    guides(col=guide_legend(nrow=20, byrow=TRUE, override.aes=list(size=3)))
        if (nKeys <= NCBPALETTE) {
            p <- p + scale_colour_manual(values=CBPALETTE)
        }
        print(p)
        dev.off()
    }
}

scatterPlot <- function(df, f, title, xlabel, ylabel, gvis) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In scatterPlot')
  }

    if(gvis == 1){
      title <- sub("\n", " ", title)
      dt <- df
      #de <- as.data.frame(cast(df, serial + x + y ~ key, value='key'))
      df$key <- NULL
      df$x <- NULL
      df$serial <- as.integer(df$serial)
      df <- df[,c("serial", "y")]
      df$y.html.tooltip <- paste(dt$key,": ", dt$y, " (", dt$x, ")",  sep="")
      serial_min <- min(df$serial, na.rm = TRUE) - 1
      serial_max <- max(df$serial, na.rm = TRUE) + 1
      p <- gvisScatterChart(df, 
                        options=list(legend="none", title=title, height=600, width=940, tooltip="{isHtml:'true'}, {textStyle: {fontSize: '18'}}",
                                     vAxis=paste("{title:'",ylabel,"',textStyle:{fontSize:'10'}}", sep=""), 
                                     hAxis=paste("{format:'', title:'",xlabel,"',textStyle:{fontSize:'14'}, viewWindow:{min:'",serial_min,"', max:'",serial_max,"'}}", sep=""), 
                                     #hAxis=paste("{format:'', title:'",xlabel,"',textStyle:{fontSize:'14'}, ticks:[2014061900, 2014061901, 2014061902, 2014061903]}", sep=""),
                                     chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
      cat(p$html$chart,file=f)
    }
    else {
      png(f, width = W, height = H)
        df$key <- NULL
        df$x <- NULL
      p <- ggplot(data=df, aes(x=serial, y=y)) +
                  geom_point(size = 4, stat="identity") +
                  labs(title=title, x=xlabel, y=ylabel) +
                  theme_bw() + scale_y_continuous(expand=c(0.05,0), labels = comma) +
                  theme(panel.grid.major.y = element_line(colour = GRIDGREY), 
                        panel.grid.minor.y = element_line(colour = GRIDGREY, linetype = "dotted"), 
                        panel.grid.major.x = element_blank(), panel.grid.minor.x = element_blank())

      print(p)
      dev.off()
    }
}


geomap <- function(df, f, title, xlabel, ylabel, gvis) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In geomap')
    msg <- paste('logger -p user.notice file =', f)
    system(msg)
  }
  if(gvis == 1){
    title <- sub("\n", " ", title)
    p <- gvisMap(df, locationvar='location', tipvar='sum', 
                 options=list(colorAxis="{colors:['#FF0000', '#00FF00']}", displayMode='regions', height=600, width=940,
                              chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
    cat(p$html$chart,file=f)
  }
}

geochart <- function(df, f, title, xlabel, ylabel, gvis) {

  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In geomap')
    msg <- paste('logger -p user.notice file =', f)
    system(msg)
  }
  if(gvis == 1){
    title <- sub("\n", " ", title)
    p <- gvisGeoChart(df, locationvar='location', sizevar='count', colorvar='color', hovervar='type',
                      options=list(displayMode='markers', height=600, width=940, colorAxis="{colors:['red','green']}", legend="'none'", tooltip="{showColorCode:false}",
                                   chartArea="{left:80,top:50,width:\"80%\",height:\"80%\"}"))
    cat(p$html$chart,file=f)
  }
}

getStmntParameters <- function(dsccon, dbdrv, dd_pltid, prepStmtNm, srvrid, start, stop) {

  # Retrieving the datasets ids for the plot
  if (!(prepStmnt("getdatasetids", dsccon))) {
    return
  }

  sql <- paste("EXECUTE ", "getdatasetids", "('", dd_pltid, "');", sep="")

  dataframe <- dbGetDataFrame(dbdrv, dsccon, dbconstr, sql)

  if (is.null(dataframe) || ncol(dataframe) != 1) {
    return
  }

  # Preparing the statement parameters with 1 or 2 datasets for the plot
  ds_ids_list <- gsub("[{}]","",dataframe[1])
  list <- strsplit(ds_ids_list,",")
  num_ds_ids <- length(list[[1]])
   
  if ( num_ds_ids == 1 ) {
    sql <- paste("EXECUTE ",prepStmtNm, "(", srvrid, ", '", list[[1]][1], "', timestamptz '", start, "', timestamptz '", stop, "');", sep="")
  }
  else if ( num_ds_ids == 2 ) {
    sql <- paste("EXECUTE ",prepStmtNm, "(", srvrid, ", '", list[[1]][1], "', '", list[[1]][2], "', timestamptz '", start, "', timestamptz '", stop, "');", sep="")
  }
  else {
    return
  }    
  return(sql)
}

generateYaml <- function(dsccon) {

  if (hh_debug) {
      system('logger -p user.notice Hedgehog: In generateYaml')
    }

  # Only allow yaml calls from local host in this implementation
  # as this is only called from the cron script for now
    if (SERVER$hostname != "localhost") 
    return(FALSE)
    
  # Determine the plot being requested
    prepStmnt("getpltdetails", dsccon)
    pltdetails <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getpltdetails(", GET$pltid, ");", sep=""))
    
    if (is.null(pltdetails)) {
        return(FALSE)
    }   
  pltnm     <- pltdetails$name

    # Nasty hard coding because the traffic size plot is split into two
  if (pltnm == 'traffic_sizes_small') {
    pltnm = 'traffic_sizes'
  } 
  if (pltnm == 'traffic_sizes_big') {
    return(TRUE)
  } 
  
    prepStmnt("getsrvr_display_name_from_id", dsccon)
    server_display_name <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getsrvr_display_name_from_id(", GET$svrid, ");", sep=""))

  # Load the config file with the rssac spec in it 
    rssac_config = yaml.load_file(paste(hh_config$directories$web_conf, "/rssac.yaml", sep=""))

  # Define the base parameters for the yaml
  metric <- sub("_", "-", pltnm)  
  metric_data <- rssac_config$metric[[metric]]
  # If the node looks like a root server, then set the service from the yaml
  # otherwise, just use the server name
  server_name_lc <- tolower(server_display_name)
  server_name_root <- substr(server_name_lc,2,6)
  if (server_name_root == "-root") {
    server_service <- paste(substr(server_name_lc,1,1), metric_data$service, sep="")
  } else {
    server_service <- server_name_lc
  }
    date <- format(as.POSIXct(sub("T", " ", GET$start)), '%Y-%m-%d')

  # Set up the output directory and file name
    main_dir <- paste(hh_config$directories$rssac, "/", format(as.POSIXct(date), '%Y'), "/", format(as.POSIXct(date), '%m'), "/",  metric, "/", sep="")
    dir.create(file.path(main_dir), showWarnings = FALSE, recursive = TRUE)
    yaml_file <- paste(main_dir, server_name_lc, "-", format(as.POSIXct(date), '%Y%m%d'), "-", metric, ".yaml", sep="")

    # Generate the query catering for the nodes specified
    prepStmntNm <- pltnm
    if (GET$ndarr == '-1') {
        prepStmntNm <- paste(prepStmntNm, "_all_nodes", sep="")
    }

  sql <- getStmntParameters(dsccon, dbdrv, GET$pltid, prepStmntNm, GET$svrid, GET$start, GET$stop)
  if(is.null(sql)){
    return(FALSE)
  }
    
  if (GET$ndarr != '-1'){
        sql <- sub(");", paste(", '", GET$ndarr, "');", sep=""), sql)
    }
  
  # Get the data frame
    if (!prepStmnt(prepStmntNm, dsccon)) 
    return(FALSE)
    
    df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, sql)

  # TODO: Handle empty data frames better
  # This simply means there isn't any data for this server on this day
  if(is.null(df) || nrow(df) == 0){
    return(TRUE)
  }

  # Now process the data from the data frame into yaml format
    yaml_out <- list('version'='rssac002v3', service=server_service, 'start-period'=paste(GET$start, ":00Z", sep=""), metric=metric)

    if (metric_data$statistics_type == "buckets") {
        de <- as.data.frame(cast(df, x ~ key, value='y', fun.aggregate=sum))  
        
        for (s in metric_data$statistics) {
            bucketlist <- list()
            
            for (i in seq(s$bucket_min, s$bucket_max, by=s$bucket_step)) {
                bucket <- paste(i, "-", sep="")
                
                if (i != s$bucket_max) {
                    bucket <- paste(bucket, i+s$bucket_step-1, sep="")
                    value <- de[which(de$x == i + (s$bucket_step - 1)/2),s$name]
                } 
                else if (max(de$x) >= s$bucket_max) {
                    value <- sum(subset(de, x >= s$bucket_max, select = s$name))
                }

                if (length(value)==0 || is.na(value) || is.null(value) || value==0) {
                    bucketlist[[bucket]] <- as.integer(0)
                } 
                else {
                    bucketlist[[bucket]] <- as.numeric(value)
                }
            }  
            yaml_out[[s$name]] <- bucketlist
        }
    }
  else if (metric_data$statistics_type == "kv_pair") {
        de <- as.data.frame(cast(df, ~ key, value='y', fun.aggregate=sum))
        for (s in metric_data$statistics) {
            value <- as.integer(0)            
            if (s %in% colnames(de)){
                value <- de[[s]]
            } 
            yaml_out[[s]] <- value
        }
    } 
  else if (metric_data$statistics_type == "kv_pair_with_alias") {
        for (s in metric_data$statistics) {
            value <- as.integer(0)
            if (s$alias %in% df$x) {
              value <- df[df$x == s$alias,2]
      }
            yaml_out[[s$name]] <- value
        }
  } 
  else if (metric_data$statistics_type == "count") {
    de <- as.data.frame(cast(df, ~ key, value='y', fun.aggregate=sum))
    
    for (s in metric_data$statistics) { 
      countlist <- list()
      for (i in colnames(de)) {
        if (i != "value") {
          countlist[[i]] <- de[[i]]
        }
      }
      yaml_out[[s$name]] <- countlist
    }
  }
  else if (metric_data$statistics_type == "average") {
    df$x <- NULL
    #de <- as.data.frame(cast(df, ~ serial, value='y', fun.aggregate=max))
    de <- as.data.frame(cast(df, ~ serial, value='y', function(x) quantile(x,c(0.95))))
    
    for (s in metric_data$statistics) { 
      countlist <- list()
      for (i in colnames(de)) {
        if (i != "value") {
          countlist[[i]] <- as.integer(de[[i]])
        }
      }
      yaml_out[[s$name]] <- countlist
    }
  }
  else if (metric_data$statistics_type == "value") {
    df$d.starttime <- NULL
    for (s in metric_data$statistics) { 
      countlist <- list()
      for(i in 1:nrow(df))  {
          countlist[df$x[[i]]] <- df$y[[i]]
      }
      yaml_out[[s$name]] <- countlist
    }
  }
  
  # Write the result to the yaml file
  # A large precision is needed to deal with large (up to 64 bit) numbers
    write(paste("---\n", gsub("\\.0\\n","\\\n", as.yaml(yaml_out,precision = 20)),sep=""), yaml_file)
    return(TRUE)
}

# initialise plot options
initPlotOptions <- function() {
  
  if (hh_debug) {
    system('logger -p user.notice Hedgehog: In initPlotOptions')
  }

  # first, create the groups that link plots to the prepared statements
  f1                      <<- c("edns_version", "do_bit", "client_port_range", "client_subnet2_trace")
  f1lookupcodes           <<- c("qtype", "rcode", "dnssec_qtype")
  f1lookupcodesnoquery    <<- c("opcode")
  f1noclr                 <<- c("rd_bit")
  f1count                 <<- c("client_subnet2_count")
  f1nonormal              <<- c("idn_qname")
  format1                 <<- c(f1, f1lookupcodes, f1lookupcodesnoquery, f1noclr, f1count, f1nonormal)
    
  f2mergekeys             <<- c("direction_vs_ipproto")
  f2mergekeys_lookup      <<- c("certain_qnames_vs_qtype")
  f2mergekeys_lookup_key1 <<- c("chaos_types_and_names")
  f2other                 <<- c("server_addr_vs_trans")
  f2sumkey2values         <<- c("pcap_stats", "transport_vs_qtype", "dns_ip_version")
  format2                 <<- c(f2mergekeys, f2mergekeys_lookup, f2mergekeys_lookup_key1, f2sumkey2values)
    
  format3                 <<- c("client_subnet_accum", "ipv6_rsn_abusers_accum")
  format3_bgpprefix       <<- c("bgpprefix")
  format3_asn             <<- c("asn")
  formattraffic           <<- c("traffic_volume", "traffic_volume_difference")

  formatother             <<- c("qtype_vs_tld", "qtype_vs_legacygtld", "qtype_vs_cctld", "qtype_vs_newgtld", "qtype_vs_othertld", "client_addr_vs_rcode_accum", "qtype_vs_qnamelen", 
                                "rcode_vs_replylen", "rcode_vs_replylen_big", "client_subnet2_accum", "dns_ip_version_vs_qtype", "by_node", "by_instance", "by_city", "by_country",
                                "load_time", "zone_size", "server_addr", "client_subnet_vs_tld", "by_region")

  rssac                   <<- c("traffic_volume", "traffic_sizes_small","traffic_sizes_big", "rcode_volume", "unique_sources", "traffic_volume_difference")
  
  geo                     <<- c("geomap", "geochart")
  
  formatother             <<- c(formatother, rssac, geo)
  formatother             <<- c(formatother, "client_addr_vs_rcode_accum_asn", "client_addr_vs_rcode_accum_bgpprefix", "client_subnet2_accum_asn", "client_subnet2_accum_bgpprefix", "dnskey_vs_asn")

  unknown_graphs          <<- c("client_subnet_count", "idn_vs_tld", "ipv6_rsn_abusers_count")

  # now create other useful groups    
  passplotname            <<- c(f1lookupcodes, f1lookupcodesnoquery)
  avgoverwindow           <<- c(format3, format3_bgpprefix, format3_asn, 'qtype_vs_tld', 'qtype_vs_legacygtld', 'qtype_vs_cctld', 'qtype_vs_newgtld', 'qtype_vs_othertld',
                                'client_addr_vs_rcode_accum', 'client_subnet2_accum', 'dns_ip_version_vs_qtype', 'client_addr_vs_rcode_accum_asn', 'client_addr_vs_rcode_accum_bgpprefix',
                                "client_subnet2_accum_asn", "client_subnet2_accum_bgpprefix", "client_subnet_vs_tld", "dnskey_vs_asn")
  lineplots               <<- c(format1, format2, "by_node", "by_instance", "by_city", "by_country", "rcode_volume", "server_addr", "by_region")
  facetedbarplots         <<- c("traffic_sizes_small","traffic_sizes_big")
  facetedlineplots        <<- c("traffic_volume")
  facetedlineplots2       <<- c("server_addr_vs_trans")
  faceteddifflineplots    <<- c("traffic_volume_difference")
  log_option              <<- c(f1, f1lookupcodes, f1lookupcodesnoquery, f1noclr, f1nonormal, format2, "by_node", "by_instance", "by_city", "by_country", "rcode_volume", "server_addr", "by_region")
}

# create plot file if not cached
generatePlotFile <- function(plttitle, pltnm, ddpltid, plot_file, simple_start, simple_stop, svrid, pltid, gvis, ndarr, dsccon) {
  
  if (hh_debug) {
    msg <- paste('logger -p user.notice In generatePlotFile. Plot name =', pltnm, sep=" ")
    system(msg)
  }
  if (pltnm == 'geomap' && gvis == 0) {
    plot_file <- "plots/no_graph.png"
    return(plot_file)
  }
  if (pltnm == 'geochart' && gvis == 0) {
    plot_file <- "plots/no_graph.png"
    return(plot_file)
  }    
    
  mytitle <- paste(plttitle, "\nfrom ", simple_start, " UTC to ", simple_stop," UTC ", sep="")
  time_window <- difftime(simple_stop, simple_start, units="secs")
      
  if (time_window > 86340 && pltnm %in% lineplots) {
    mytitle <- paste(mytitle, " (smoothed)", sep="")
  }

  # Choose the prepared statement based on the group
  prepStmntNm <- ""
  if      (pltnm %in% f1)                         {prepStmntNm <- "f1"}
  else if (pltnm %in% f1lookupcodes )             {prepStmntNm <- "f1lookupcodes"}
  else if (pltnm %in% f1lookupcodesnoquery)       {prepStmntNm <- "f1lookupcodesnoquery"}
  else if (pltnm %in% f1noclr)                    {prepStmntNm <- "f1noclr"}
  else if (pltnm %in% f1count)                    {prepStmntNm <- "f1count"}
  else if (pltnm %in% f1nonormal)                 {prepStmntNm <- "f1nonormal"}
  else if (pltnm %in% f2mergekeys)                {prepStmntNm <- "f2mergekeys"}
  else if (pltnm %in% f2mergekeys_lookup)         {prepStmntNm <- "f2mergekeys_lookup"}
  else if (pltnm %in% f2mergekeys_lookup_key1)    {prepStmntNm <- "f2mergekeys_lookup_key1"}
  else if (pltnm %in% f2sumkey2values)            {prepStmntNm <- "f2sumkey2values"}
  else if (pltnm %in% f2other)                    {prepStmntNm <- "f2mergekeys"}
  else if (pltnm %in% format3 )                   {prepStmntNm <- "format3"}
  else if (pltnm %in% format3_bgpprefix )         {prepStmntNm <- "format3_bgpprefix"}
  else if (pltnm %in% format3_asn )               {prepStmntNm <- "format3_asn"}  
  else if (pltnm %in% formattraffic )             {prepStmntNm <- "traffic_volume"}
  else if (pltnm %in% formatother)                {prepStmntNm <-  pltnm}

  # Do other fix ups to the SQL before running the query
  if (ndarr == '-1') {
    prepStmntNm <- paste(prepStmntNm, "_all_nodes", sep="")
  }
  
  if (!(prepStmnt(prepStmntNm, dsccon))) {
    return
  }
  # Setting statement parameters with 1 or 2 datasets for the plot
  sql <- getStmntParameters(dsccon, dbdrv, ddpltid, prepStmntNm, svrid, simple_start, simple_stop)
  if (is.null(sql)) {
    return
  } 
  if (pltnm %in% passplotname){
    sql <- sub(");", paste(", '", pltnm, "');", sep=""), sql)
  }
  if (ndarr != '-1'){
    sql <- sub(");", paste(", '", ndarr, "');", sep=""), sql)
  }
  if(pltnm %in% avgoverwindow){
    sql <- sub("\\(", paste("\\(" , time_window, ".0, ", sep=""), sql)
  }

  # Get the data  **RUN THE QUERY **
  df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, sql)

    # Set the axis labels
  xlab <- "Date"
  if       (pltnm %in% format3)                   {xlab <- "Subnet (IPv4/8 or IPv6/32)"}
  else if  (pltnm %in% format3_bgpprefix)         {xlab <- "Subnet (As seen in BGP)"}
  else if  (pltnm %in% format3_asn)               {xlab <- "Subnet ASN"}
  else if  (pltnm == 'traffic_sizes_small' ||
            pltnm == 'traffic_sizes_big')         {xlab <- "Message Size (bytes)"}
  else if (pltnm == 'zone_size')              {xlab <- "Serial Number"}
  else if (pltnm == 'load_time')              {xlab <- "Serial Number"}
    
  ylab <- "Queries/sec"
  if      (pltnm %in% rssac)                  {ylab <- "Queries/min"}
  else if (pltnm %in% f1count)                {ylab <- "# Client Subnets"}
  else if (pltnm %in% format3 ||
           pltnm %in% format3_bgpprefix ||
           pltnm %in% format3_asn)                {ylab <- "Average Query Rate (q/sec)"}
  else if (pltnm == 'traffic_sizes_small' ||
             pltnm == 'traffic_sizes_big')      {ylab <- "Number of Queries in Each 16 Byte Group"}
  else if (pltnm == 'zone_size')              {ylab <- "Zone Size (Bytes)"}
  else if (pltnm == 'load_time')              {ylab <- "Zone Propagation Time (Seconds)"}
    
  # Now decide how to plot it
  if      (is.null(df))                               {plot_file <- "plots/no_connection.png"}
  else if (nrow(df) == 0)                             {plot_file <- "plots/no_results.png"}
  else if ((nrow(df) == 1) && (pltnm %in% lineplots)) {plot_file <- "plots/no_results.png"}
  else if (pltnm %in% f1count)                        {stackedAreaPlot     (df, plot_file, mytitle, xlab, ylab, gvis)}
  else if (pltnm %in% lineplots)                      {linePlot            (df, plot_file, mytitle, xlab, ylab, gvis)}
  else if (pltnm %in% facetedlineplots)               {facetedLinePlot     (df, plot_file, mytitle, xlab, ylab, gvis)}
  else if (pltnm %in% facetedlineplots2)              {facetedLinePlot2    (df, plot_file, mytitle, xlab, ylab, gvis)}
  else if (pltnm %in% faceteddifflineplots)           {facetedDiffLinePlot (df, plot_file, mytitle, xlab, ylab, gvis)}
  else if (pltnm %in% facetedbarplots)                {facetedBarPlot      (df, plot_file, mytitle, xlab, ylab, gvis, 14)} # width hardcoded to 14
  else if (pltnm %in% format3 ||
           pltnm %in% format3_bgpprefix ||
           pltnm %in% format3_asn)                        {barPlot             (df, plot_file, mytitle, xlab, ylab, gvis)}
  else if (pltnm == 'qtype_vs_tld' || 
           pltnm == 'qtype_vs_legacygtld' || 
           pltnm == 'qtype_vs_cctld' || 
           pltnm == 'qtype_vs_newgtld' ||
           pltnm == 'qtype_vs_othertld')             {stackedBarPlot(df, plot_file, mytitle, "TLD",                        "Average Query Rate (q/sec)", gvis, pltnm)}
  else if (pltnm == 'client_addr_vs_rcode_accum' || 
           pltnm == 'client_subnet2_accum' ||
           pltnm == 'client_subnet_vs_tld')           {stackedBarPlot(df, plot_file, mytitle, "Subnet (IPv4/8 or IPv6/32)", "Average Query Rate (q/sec)", gvis, pltnm)}
  else if (pltnm == 'client_addr_vs_rcode_accum_asn') {stackedBarPlot(df, plot_file, mytitle, "Subnet by ASN", "Average Query Rate (q/sec)", gvis, pltnm)}
  else if (pltnm == 'client_addr_vs_rcode_accum_bgpprefix') {stackedBarPlot(df, plot_file, mytitle, "Subnet as seen in BGP", "Average Query Rate (q/sec)", gvis, pltnm)}
  else if (pltnm == 'client_subnet2_accum_asn')       {stackedBarPlot(df, plot_file, mytitle, "Subnet by ASN", "Average Query Rate (q/sec)", gvis, pltnm)}
  else if (pltnm == 'client_subnet2_accum_bgpprefix')       {stackedBarPlot(df, plot_file, mytitle, "Subnet as seen in BGP", "Average Query Rate (q/sec)", gvis, pltnm)}

  else if (pltnm == 'dnskey_vs_asn')                  {stackedBarPlot(df, plot_file, mytitle, "Count of DNSKEY Queries by ASN", "Average Query Rate (q/sec)", gvis, pltnm)}
    
  else if (pltnm == 'qtype_vs_qnamelen')              {stackedBarPlot(df, plot_file, mytitle, "QNAME Length (bytes)",       "Count",                      gvis, pltnm, scalex="continuous", vertical=1)}

  else if (pltnm == 'dns_ip_version_vs_qtype')        {stackedBarPlot(df, plot_file, mytitle, "IP Version",                 "Average Query Rate (q/sec)", gvis, pltnm, vertical=1)}
  else if (pltnm == 'unique_sources')                 {barPlot       (df, plot_file, mytitle, "IP Version/Aggregation",     "Number of Unique Sources",   gvis, small=0)}
  else if (pltnm == 'rcode_vs_replylen' || 
           pltnm == 'rcode_vs_replylen_big')          {if (gvis==1) {stackedAreaPlot(df, plot_file, mytitle, "Response Size Length (bytes)", "Count", gvis)}
                                                       else         {stackedBarPlot (df, plot_file, mytitle, "Response Size Length (bytes)", "Count", gvis, pltnm, scalex="continuous", vertical=1)}}
  else if (pltnm == 'geomap')                         {geomap  (df, plot_file, mytitle, "aaaa", "bbbbb", 1)}
  else if (pltnm == 'geochart')                       {geochart(df, plot_file, mytitle, "aaaa", "bbbbb", 1)}
  else if (pltnm == 'load_time')                      {scatterPlot(df, plot_file, mytitle, xlab, ylab, gvis)}
  else if (pltnm == 'zone_size')                      {barPlot(df, plot_file, mytitle, xlab, ylab, gvis, vertical=1)}

  return(plot_file)
}
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

#TODO(refactor): Tidy this up so common strings are re-used and we don't need separate node and all_node versions of the same query

prepStmnt <- function(statementNm, dsccon){

    if(class(dsccon) != "try-error"){
      select_nodes_sql="AND node_id = ANY (string_to_array($6, ',')::integer[])"
      all_nodes_prep="(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS"
      select_nodes_prep="(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS"
      skipped_sql="NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')"
      
        rs <- NULL
        switch(statementNm,
               getdbversion                         = { rs <- try(dbSendQuery(dsccon, "PREPARE getdbversion AS                      SELECT version FROM dsc.version;"))},
               getpltddcategories                   = { rs <- try(dbSendQuery(dsccon, "PREPARE getpltddcategories AS                SELECT DISTINCT ddcategory, cast(substring(ddcategory from 1 for position('.' in ddcategory)-1 ) as int) as in_order FROM dsc.plot WHERE ddcategory != '' order by in_order;"))},
               getpltid_ddname                      = { rs <- try(dbSendQuery(dsccon, "PREPARE getpltid_ddname (TEXT) AS            SELECT id, ddname FROM dsc.plot WHERE ddcategory=$1 ORDER BY ddname ;"))},
               getpltdetails                        = { rs <- try(dbSendQuery(dsccon, "PREPARE getpltdetails (INTEGER) AS           SELECT name, title, description, plot_id FROM dsc.plot WHERE id=$1;"))},
               getsrvrid_display_name               = { rs <- try(dbSendQuery(dsccon, "PREPARE getsrvrid_display_name AS            SELECT id, display_name FROM dsc.server ORDER BY display_name;"))},
               getsrvr_display_name_from_id         = { rs <- try(dbSendQuery(dsccon, "PREPARE getsrvr_display_name_from_id AS      SELECT display_name FROM dsc.server where id=$1;"))},
               getgroups                            = { rs <- try(dbSendQuery(dsccon, "PREPARE getgroups (INTEGER) AS               SELECT DISTINCT CASE WHEN (region='' OR region IS NULL) THEN 'Other' ELSE region END FROM dsc.node WHERE server_id=$1 ORDER BY region;"))},

               getnodes                             = { rs <- try(dbSendQuery(dsccon, "PREPARE getnodes         (INTEGER) AS        SELECT id, name, CASE WHEN (country='' OR country IS NULL) THEN 'No Country' ELSE country END as country, CASE WHEN (city='' OR city IS NULL) THEN 'No City' ELSE city END as city, CASE WHEN (subgroup='' OR subgroup IS NULL) THEN 'No Instance' ELSE subgroup END as subgroup FROM dsc.node WHERE server_id=$1 ORDER BY name;"))},
               getnodesbyregion                     = { rs <- try(dbSendQuery(dsccon, "PREPARE getnodesbyregion (INTEGER, TEXT) AS  SELECT id, name, CASE WHEN (country='' OR country IS NULL) THEN 'No Country' ELSE country END as country, CASE WHEN (city='' OR city IS NULL) THEN 'No City' ELSE city END as city, CASE WHEN (subgroup='' OR subgroup IS NULL) THEN 'No Instance' ELSE subgroup END as subgroup FROM dsc.node WHERE server_id=$1 AND region=$2 ORDER BY name;"))},

               getdatasetids                        = { rs <- try(dbSendQuery(dsccon, "PREPARE getdatasetids    (INTEGER) AS        SELECT dataset_id FROM dsc.plot WHERE id=$1;"))},
               getdefaultpltid                      = { rs <- try(dbSendQuery(dsccon, "PREPARE getdefaultpltid  (TEXT) AS           SELECT id FROM dsc.plot WHERE name=$1;"))},

               by_node                              = { rs <- try(dbSendQuery(dsccon, "PREPARE by_node                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, n.name AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_node_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE by_node_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, n.name AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},

               by_region                            = { rs <- try(dbSendQuery(dsccon, "PREPARE by_region                            (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.region   = '' OR n.region   IS NULL) THEN 'Other'       ELSE n.region   END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_region_all_nodes                  = { rs <- try(dbSendQuery(dsccon, "PREPARE by_region_all_nodes                  (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.region   = '' OR n.region   IS NULL) THEN 'Other'       ELSE n.region   END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_instance                          = { rs <- try(dbSendQuery(dsccon, "PREPARE by_instance                          (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.subgroup = '' OR n.subgroup IS NULL) THEN 'No Instance' ELSE n.subgroup END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_instance_all_nodes                = { rs <- try(dbSendQuery(dsccon, "PREPARE by_instance_all_nodes                (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.subgroup = '' OR n.subgroup IS NULL) THEN 'No Instance' ELSE n.subgroup END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_city                              = { rs <- try(dbSendQuery(dsccon, "PREPARE by_city                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.city     = '' OR n.city     IS NULL) THEN 'No City'     ELSE n.city     END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_city_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE by_city_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.city     = '' OR n.city     IS NULL) THEN 'No City'     ELSE n.city     END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_country                           = { rs <- try(dbSendQuery(dsccon, "PREPARE by_country                           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.country  = '' OR n.country  IS NULL) THEN 'No Country'  ELSE n.country  END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               by_country_all_nodes                 = { rs <- try(dbSendQuery(dsccon, "PREPARE by_country_all_nodes                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN (n.country  = '' OR n.country  IS NULL) THEN 'No Country'  ELSE n.country  END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},

               f1                                   = { rs <- try(dbSendQuery(dsccon, "PREPARE f1                                   (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1_all_nodes                         = { rs <- try(dbSendQuery(dsccon, "PREPARE f1_all_nodes                         (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1lookupcodes                        = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodes                        (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT, TEXT) AS     select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IN ('FormErr', 'NotImp', 'YXDomain', 'YXRRSet', 'NotAuth', 'NotZone', 'BADKEY', 'BADTIME', 'BADMODE', 'BADNAME', 'BADALG', 'BADTRUNC', 'BADVERS') THEN 'Other' WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1lookupcodes_all_nodes              = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodes_all_nodes              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IN ('FormErr', 'NotImp', 'YXDomain', 'YXRRSet', 'NotAuth', 'NotZone', 'BADKEY', 'BADTIME', 'BADMODE', 'BADNAME', 'BADALG', 'BADTRUNC', 'BADVERS') THEN 'Other' WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1lookupcodesnoquery                 = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodesnoquery                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) AND d.key1 !='Query' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1lookupcodesnoquery_all_nodes       = { rs <- try(dbSendQuery(dsccon, "PREPARE f1lookupcodesnoquery_all_nodes       (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key1::text = il.value::text AND il.registry::text = $5 WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.key1 !='Query' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1noclr                              = { rs <- try(dbSendQuery(dsccon, "PREPARE f1noclr                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND key1 !='clr' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1noclr_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE f1noclr_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1 !='clr' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1count                              = { rs <- try(dbSendQuery(dsccon, "PREPARE f1count                              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value) AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1count_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE f1count_all_nodes                    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value) AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1nonormal                           = { rs <- try(dbSendQuery(dsccon, "PREPARE f1nonormal                           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND key1 !='normal' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f1nonormal_all_nodes                 = { rs <- try(dbSendQuery(dsccon, "PREPARE f1nonormal_all_nodes                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1 !='normal' GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2mergekeys                          = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys                          (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2mergekeys_all_nodes                = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_all_nodes                (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2mergekeys_lookup                   = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup                   (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry::text='qtype' WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2mergekeys_lookup_all_nodes         = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup_all_nodes         (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, d.key1 || ':' || CASE WHEN il.name IS NULL then 'Other' ELSE il.name END AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry::text='qtype' WHERE d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2mergekeys_lookup_key1              = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup_key1              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, il.name || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, iana_lookup il WHERE il.registry::text='qtype' AND key1::text = il.value::text AND d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2mergekeys_lookup_key1_all_nodes    = { rs <- try(dbSendQuery(dsccon, "PREPARE f2mergekeys_lookup_key1_all_nodes    (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT d.starttime AS sx, il.name || ':' || key2 AS skey, sum(d.value)/60.0 AS sy FROM dsc.data d, iana_lookup il WHERE il.registry::text='qtype' AND key1::text = il.value::text AND d.key1 != 'else' AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2sumkey2values                      = { rs <- try(dbSendQuery(dsccon, "PREPARE f2sumkey2values                      (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               f2sumkey2values_all_nodes            = { rs <- try(dbSendQuery(dsccon, "PREPARE f2sumkey2values_all_nodes            (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 select to_timestamp((extract(epoch from sq.sx)::int / (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*(extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x, sq.skey as key, avg(sq.sy) as y from (SELECT starttime   AS sx, key1 AS skey, sum(value)/60.0 AS sy FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY sx, skey) as sq GROUP BY x, key;"))},
               format3                              = { rs <- try(dbSendQuery(dsccon, "PREPARE format3                        (REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS           SELECT dsc.iptruncate(key2::ipaddress) AS x, sum(value)/$1 AS y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND key2 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x ORDER BY y DESC LIMIT 40;"))},
               format3_all_nodes                    = { rs <- try(dbSendQuery(dsccon, "PREPARE format3_all_nodes              (REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                 SELECT dsc.iptruncate(key2::ipaddress) AS x, sum(value)/$1 AS y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x ORDER BY y DESC LIMIT 40;"))},

               client_addr_vs_rcode_accum           = { rs <- try(dbSendQuery(dsccon, "PREPARE client_addr_vs_rcode_accum               (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS  SELECT dsc.iptruncate(key1::ipaddress) AS x, il.name AS key, sum(d.value)/$1 As y FROM dsc.data d, iana_lookup il WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'rcode' AND d.key2::text = il.value::text AND dsc.iptruncate(key1::ipaddress) IN (SELECT dsc.iptruncate(key1::ipaddress) AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
               client_addr_vs_rcode_accum_all_nodes = { rs <- try(dbSendQuery(dsccon, "PREPARE client_addr_vs_rcode_accum_all_nodes     (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ) AS        SELECT dsc.iptruncate(key1::ipaddress) AS x, il.name AS key, sum(d.value)/$1 As y FROM dsc.data d, iana_lookup il WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'rcode' AND d.key2::text = il.value::text AND dsc.iptruncate(key1::ipaddress) IN (SELECT dsc.iptruncate(key1::ipaddress) AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
               qtype_vs_qnamelen                    = { rs <- try(dbSendQuery(dsccon, "PREPARE qtype_vs_qnamelen                     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT il.name AS key, cast(d.key2 AS int) AS x, sum(d.value) AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'qtype' AND d.key1::text = il.value::text AND cast(d.key2 AS int) < 100 GROUP BY key, x;"))},
               qtype_vs_qnamelen_all_nodes          = { rs <- try(dbSendQuery(dsccon, "PREPARE qtype_vs_qnamelen_all_nodes           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT il.name AS key, cast(d.key2 AS int) AS x, sum(d.value) AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND il.registry = 'qtype' AND d.key1::text = il.value::text AND cast(d.key2 AS int) < 100 GROUP BY key, x;"))},
               rcode_vs_replylen                    = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen                     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer < 1000;"))},
               rcode_vs_replylen_all_nodes          = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen_all_nodes           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer < 1000;"))},
               rcode_vs_replylen_big                = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen_big                 (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND node_id = ANY (string_to_array($5, ',')::integer[]) AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer >= 1000;"))},
               rcode_vs_replylen_big_all_nodes      = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_vs_replylen_big_all_nodes       (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT il.name AS key, key2::integer AS x, d.value AS y FROM dsc.data d, iana_lookup il WHERE server_id=$1 AND d.plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND il.registry = 'rcode' AND d.key1::text = il.value::text AND key2::integer >= 1000;"))},
               client_subnet2_accum                 = { rs <- try(dbSendQuery(dsccon, "PREPARE client_subnet2_accum                     (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS  SELECT dsc.iptruncate(key1::ipaddress) AS x, key2 AS key, sum(value)/$1 As y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND substring(key1 FROM '^([0-9]*.)|.*:.*') IN (SELECT substring(key1 FROM '^([0-9]*.)|.*:.*')::text AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[]) AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
               client_subnet2_accum_all_nodes       = { rs <- try(dbSendQuery(dsccon, "PREPARE client_subnet2_accum_all_nodes           (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ) AS        SELECT dsc.iptruncate(key1::ipaddress) AS x, key2 AS key, sum(value)/$1 As y FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND substring(key1 FROM '^([0-9]*.)|.*:.*') IN (SELECT substring(key1 FROM '^([0-9]*.)|.*:.*')::text AS k1 FROM dsc.data WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40) GROUP BY x, key;"))},
               dns_ip_version_vs_qtype              = { rs <- try(dbSendQuery(dsccon, "PREPARE dns_ip_version_vs_qtype                  (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS  SELECT d.key1 AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 As y FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype' WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x, key ORDER BY y DESC;"))},
               dns_ip_version_vs_qtype_all_nodes    = { rs <- try(dbSendQuery(dsccon, "PREPARE dns_ip_version_vs_qtype_all_nodes        (REAL, INTEGER, INTEGER,     TIMESTAMPTZ, TIMESTAMPTZ) AS        SELECT d.key1 AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 As y FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype' WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') GROUP BY x, key ORDER BY y DESC;"))},

               traffic_volume                       = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_volume                   (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT starttime AS x, 'dns-' || key1 || '-queries-received-' || lower(key2) AS key, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY x, key UNION SELECT starttime AS x, 'dns-' || key1 || '-responses-sent-' || lower(key2) AS key, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY x, key;"))},
               traffic_volume_all_nodes             = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_volume_all_nodes         (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT starttime AS x, 'dns-' || key1 || '-queries-received-' || lower(key2) AS key, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 GROUP BY x, key UNION SELECT starttime AS x, 'dns-' || key1 || '-responses-sent-' || lower(key2) AS key, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 GROUP BY x, key;"))},
               # Following two statements are used to generate yaml data, but not for a visible plot
               traffic_sizes                        = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes                    (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5  AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x ORDER BY x;"))},
               traffic_sizes_all_nodes              = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_all_nodes          (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data   WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 GROUP BY key, x ORDER BY x;"))},
               traffic_sizes_small                  = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_small              (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x ORDER BY x;"))},
               traffic_sizes_small_all_nodes        = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_small_all_nodes    (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data   WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer < 1000 GROUP BY key, x ORDER BY x;"))},
               traffic_sizes_big                    = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_big                (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS      SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data d WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 AND d.node_id = ANY (string_to_array($6, ',')::integer[]) GROUP BY key, x ORDER BY x;"))},
               traffic_sizes_big_all_nodes          = { rs <- try(dbSendQuery(dsccon, "PREPARE traffic_sizes_big_all_nodes      (INTEGER, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS            SELECT key1 || '-request-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data   WHERE server_id=$1 AND plot_id=$2 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 GROUP BY key, x UNION SELECT key1 || '-response-sizes' AS key, ((key2::integer/16) * 16) + 7.5 AS x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5 AND key2::integer >= 1000 GROUP BY key, x ORDER BY x;"))},
               rcode_volume                         = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_volume                     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT d.starttime AS x, d.key1 AS key, sum(d.value) AS y FROM dsc.data d WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY x, key;"))},
               rcode_volume_all_nodes               = { rs <- try(dbSendQuery(dsccon, "PREPARE rcode_volume_all_nodes           (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT d.starttime AS x, d.key1 AS key, sum(d.value) AS y FROM dsc.data d WHERE d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 GROUP BY x, key;"))},
               unique_sources_raw                   = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources_raw               (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT key1 as x, count(key2) AS y FROM dsc.data as d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY x UNION SELECT 'IPv6/64' as x, count(*) as y from (SELECT substring(key2 FROM '(^([0-9a-f]{1,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}))') as subnet FROM dsc.data as d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1='IPv6' AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY subnet) AS sq ORDER BY y DESC;"))},
               unique_sources_raw_all_nodes         = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources_raw_all_nodes     (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT key1 as x, count(key2) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY x UNION SELECT 'IPv6/64' as x, count(*) as y from (SELECT substring(key2 FROM '(^([0-9a-f]{1,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}:{0,1}[0-9a-f]{0,4}))') as subnet FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND key1='IPv6' GROUP BY subnet) AS sq ORDER BY y DESC;"))},
               unique_sources                       = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources                   (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT key1 as x, sum(value) AS y FROM dsc.data as d WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) GROUP BY x order by y DESC;"))},
               unique_sources_all_nodes             = { rs <- try(dbSendQuery(dsccon, "PREPARE unique_sources_all_nodes         (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT key1 as x, sum(value) AS y FROM dsc.data WHERE server_id=$1 AND plot_id=$2 AND starttime>=$3 AND starttime<=$4 GROUP BY x order by y desc;"))},
               load_time                            = { rs <- try(dbSendQuery(dsccon, "PREPARE load_time                        (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT d.starttime AS x, n.name AS key, d.value AS y, d.key2 as serial FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) order by serial;"))},
               load_time_all_nodes                  = { rs <- try(dbSendQuery(dsccon, "PREPARE load_time_all_nodes              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT d.starttime AS x, n.name AS key, d.value AS y, d.key2 as serial FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 order by serial;"))},
               zone_size                            = { rs <- try(dbSendQuery(dsccon, "PREPARE zone_size                        (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS               SELECT d.starttime, avg(d.value) AS y, d.key2 AS x FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) group by d.starttime, d.key2 order by x;"))},
               zone_size_all_nodes                  = { rs <- try(dbSendQuery(dsccon, "PREPARE zone_size_all_nodes              (INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS                     SELECT d.starttime, avg(d.value) AS y, d.key2 AS x FROM dsc.data d, dsc.node n WHERE d.node_id = n.id AND d.server_id=$1 AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4 group by d.starttime, d.key2 order by x;"))},

               geomap = {
                 sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS
                 select g.latitude || ':' || g.longitude as location, sum(d.value) 
                 FROM dsc.data d, geoip g where cast(d.key2 as ipaddress) <<= g.ip_range 
                 and server_id=$1 AND plot_id=$2 AND d.starttime >=$3 
                 AND starttime<=$4 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) 
                 and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
                 group by location;", sep=" ")
                sql=gsub("\n"," ",sql_joined)
                rs <- try(dbSendQuery(dsccon, sql))},
               
               geomap_all_nodes = {
                 sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS
                 select g.latitude || ':' || g.longitude as location, sum(d.value) 
                 FROM dsc.data d, geoip g 
                 where cast(d.key2 as ipaddress) <<= g.ip_range 
                 and server_id=$1 AND plot_id=$2 AND d.starttime >=$3 
                 AND starttime<=$4 and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
                 group by location;", sep=" ")
                sql=gsub("\n"," ",sql_joined)
                rs <- try(dbSendQuery(dsccon, sql))},
               
               geochart = {
                 sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS
                 select l.name as location, sum(d.value) as count, 1 as color, 'Queries' as type 
                 FROM dsc.data d, geoip g, locations l  
                 where cast(d.key2 as ipaddress) <<= g.ip_range 
                 and l.geoname_id=g.geoname and server_id=$1 AND plot_id=$2 
                 AND d.starttime >=$3 AND starttime<=$4 
                 AND d.node_id = ANY (string_to_array($5, ',')::integer[]) 
                 and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
                 group by location 
                 union 
                 select n.city as location, count(*), 2 as color, n.city || ' nodes' as type 
                 from node n 
                 where n.id = ANY (string_to_array($5, ',')::integer[])
                 and server_id=$1 and n.city!='' and n.city IS NOT NULL
                 group by n.city;", sep=" ")
                sql=gsub("\n"," ",sql_joined)
                rs <- try(dbSendQuery(dsccon, sql))},
               
               geochart_all_nodes = {
                 sql_joined <- paste("PREPARE", statementNm,  "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS
                 select l.name as location, sum(d.value) as count, 1 as color, 'Queries' as type 
                 FROM dsc.data d, geoip g, locations l  
                 where cast(d.key2 as ipaddress) <<= g.ip_range 
                   and l.geoname_id=g.geoname and server_id=$1 AND plot_id=$2 
                   AND d.starttime >=$3 AND starttime<=$4 
                   and d.key2 not in ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
                 group by location 
                 union 
                 select n.city as location, count(*), 2 as color, n.city || ' nodes' as type 
                 from node n
                 where server_id=$1 and n.city!='' and n.city IS NOT NULL
                 group by n.city;", sep=" ")
                sql=gsub("\n"," ",sql_joined)
                rs <- try(dbSendQuery(dsccon, sql))},

               format3_bgpprefix  = {
                 sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
                 "SELECT xx / ip2bgpprefix(xx) AS x, sum(y) as y FROM
                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y 
                  FROM dsc.data 
                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                  select_nodes_sql,
                  "AND key2", skipped_sql,
                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
                group by x order by y desc;", sep=" ")
                sql=gsub("\n"," ",sql_joined)
                rs <- try(dbSendQuery(dsccon, sql))},
               
               format3_bgpprefix_all_nodes= {
                 sql_joined <- paste("PREPARE", statementNm, all_nodes_prep, 
                 "SELECT xx / ip2bgpprefix(xx) AS x, sum(y) as y FROM 
                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y
                   FROM dsc.data
                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
                   AND key2", skipped_sql,
                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
                group by x order by y desc;", sep=" ")
                sql=gsub("\n"," ",sql_joined)
                rs <- try(dbSendQuery(dsccon, sql))},

               format3_asn = {
                 sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
                 "SELECT ip2asn(xx) AS x, sum(y) as y FROM
                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y 
                   FROM dsc.data 
                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                   select_nodes_sql,
                   "AND key2", skipped_sql,
                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
                group by x order by y desc;", sep=" ")
                sql=gsub("\n"," ",sql_joined)
                rs <- try(dbSendQuery(dsccon, sql))},
               
               format3_asn_all_nodes = {
                 sql_joined <- paste("PREPARE", statementNm, all_nodes_prep, 
                 "SELECT ip2asn(xx) AS x, sum(y) as y FROM 
                  (SELECT key2::ipaddress AS xx, sum(value)/$1 AS y 
                   FROM dsc.data 
                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
                   AND key2", skipped_sql,
                  "GROUP BY xx ORDER BY y DESC LIMIT 40) as sq
                group by x order by y desc;", sep=" ")
                 sql=gsub("\n"," ",sql_joined)
                 rs <- try(dbSendQuery(dsccon, sql))},
               
               client_addr_vs_rcode_accum_asn = {
               sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
                 "SELECT dsc.ip2asn(xx) AS x, key, sum(y) as y FROM 
                  (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
                   FROM dsc.data d, iana_lookup il 
                   WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
                   AND d.key1", skipped_sql,
                   "AND il.registry = 'rcode' AND d.key2::text = il.value::text",
                   select_nodes_sql,
                   "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
              sql=gsub("\n"," ",sql_joined)
              rs <- try(dbSendQuery(dsccon, sql))},
              
              client_addr_vs_rcode_accum_asn_all_nodes = {
              sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
                "SELECT dsc.ip2asn(xx) AS x, key, sum(y) as y FROM 
                 (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
                  FROM dsc.data d, iana_lookup il 
                  WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
                  AND d.key1 ", skipped_sql,
                  "AND il.registry = 'rcode' AND d.key2::text = il.value::text
                  GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
             sql=gsub("\n"," ",sql_joined)
             rs <- try(dbSendQuery(dsccon, sql))},
 
             client_addr_vs_rcode_accum_bgpprefix = {
             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
               "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM 
                (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
                 FROM dsc.data d, iana_lookup il 
                 WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
                 AND d.key1", skipped_sql,
                 "AND il.registry = 'rcode' AND d.key2::text = il.value::text",
                 select_nodes_sql,
                 "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
            sql=gsub("\n"," ",sql_joined)
            rs <- try(dbSendQuery(dsccon, sql))},
            
            client_addr_vs_rcode_accum_bgpprefix_all_nodes = {
            sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
              "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM 
               (SELECT key1::ipaddress AS xx, il.name AS key, sum(d.value)/$1 As y 
                FROM dsc.data d, iana_lookup il 
                WHERE server_id=$2 AND d.plot_id=$3 AND starttime>=$4 AND starttime<=$5
                AND d.key1", skipped_sql,
                  "AND il.registry = 'rcode' AND d.key2::text = il.value::text
                GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           client_subnet2_accum_asn = {
             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
                  FROM dsc.data 
                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                  select_nodes_sql,
                  "AND key1", skipped_sql,
                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},
           
           client_subnet2_accum_asn_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
                  FROM dsc.data 
                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                  "AND key1", skipped_sql,
                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           client_subnet2_accum_bgpprefix = {
             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
               "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM
                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
                  FROM dsc.data 
                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                  select_nodes_sql,
                  "AND key1", skipped_sql,
                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},
           
           client_subnet2_accum_bgpprefix_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
               "SELECT xx / dsc.ip2bgpprefix(xx) AS x, key, sum(y) as y FROM
                 (SELECT key1::ipaddress AS xx, key2 AS key, sum(value)/$1 AS y
                  FROM dsc.data 
                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                  "AND key1", skipped_sql,
                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},               

           server_addr = {
             sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
               "select to_timestamp((extract(epoch from sq.sx)::int /
                 (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*
                 (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x,
                sq.skey as key, avg(sq.sy) as y from
                  (SELECT d.starttime AS sx, d.key1 AS skey, sum(d.value)/60.0 AS sy
                   FROM dsc.data d
                   WHERE d.server_id=$1
                    AND d.plot_id=$2 AND d.starttime>=$3 AND d.starttime<=$4
                    AND d.node_id = ANY (string_to_array($5, ',')::integer[])
                    AND (d.key1::ipaddress IN (SELECT address FROM service_addr WHERE server_id=$1) OR (SELECT count(*) FROM service_addr WHERE server_id=$1) = 0)
                   GROUP BY sx, skey) as sq
                GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},


           server_addr_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, "(INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
             "select to_timestamp((extract(epoch from sq.sx)::int /
               (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int )*
               (extract(epoch from $4::timestamp - $3::timestamp + interval '1 minute')/1440)::int) AS x,
              sq.skey as key, avg(sq.sy) as y from
                (SELECT d.starttime AS sx, d.key1 AS skey,
                sum(d.value)/60.0 AS sy 
                FROM dsc.data d
                WHERE d.server_id=$1 AND d.plot_id=$2
                  AND d.starttime>=$3 AND d.starttime<=$4
                  AND (d.key1::ipaddress IN (SELECT address FROM service_addr WHERE server_id=$1) OR (SELECT count(*) FROM service_addr WHERE server_id=$1) = 0)
                GROUP BY sx, skey) as sq
              GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           # client_subnet_vs_tld = {
           #   sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
           #   "SELECT dsc.iptruncate(key1::ipaddress)::text AS x, key2 AS key, sum(value)/$1 As y
           #   FROM dsc.data
           #   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
           #     AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
           #     AND node_id = ANY (string_to_array($6, ',')::integer[])
           #     AND dsc.iptruncate(key1::ipaddress) IN
           #     (SELECT dsc.iptruncate(key1::ipaddress) AS k1
           #       FROM dsc.data
           #       WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4
           #       AND starttime<=$5 AND node_id = ANY (string_to_array($6, ',')::integer[])
           #       AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
           #       GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40)
           #  GROUP BY x, key;", sep=" ")
           # sql=gsub("\n"," ",sql_joined)
           # rs <- try(dbSendQuery(dsccon, sql))},
           #
           # client_subnet_vs_tld_all_nodes = {
           #   sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
           #   "SELECT dsc.iptruncate(key1::ipaddress)::text AS x, key2 AS key, sum(value)/$1 As y
           #   FROM dsc.data
           #   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
           #     AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
           #     AND dsc.iptruncate(key1::ipaddress) IN
           #     (SELECT dsc.iptruncate(key1::ipaddress) AS k1
           #       FROM dsc.data
           #       WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4
           #       AND starttime<=$5
           #       AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
           #     GROUP BY k1 ORDER BY sum(value)/$1 DESC LIMIT 40)
           #   GROUP BY x, key;", sep=" ")
           # sql=gsub("\n"," ",sql_joined)
           # rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_tld = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
             "SELECT lower(d.key1) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5
               AND d.node_id = ANY (string_to_array($6, ',')::integer[])
               AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
               AND d.key1 IN
                 (SELECT key1
                   FROM dsc.data 
                   WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
                   AND node_id = ANY (string_to_array($6, ',')::integer[])
                   AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
                   GROUP BY key1 ORDER BY sum(value)/$1 DESC LIMIT 40)
             GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_tld_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS", 
             "SELECT lower(d.key1) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') AND d.key1 IN
               (SELECT key1
                 FROM dsc.data 
                 WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5
                 AND key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
                 GROUP BY key1 ORDER BY sum(value)/$1 DESC LIMIT 40)
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_cctld = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
             "SELECT alabel2ulabel(t.alabel) as x,
             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t
             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3
             AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
             AND lower(d.key1) IN
               (SELECT lower(dd.key1)
                 FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
                   LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
                 WHERE nn.type_name='ccTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
                 AND dd.starttime>=$4 AND dd.starttime<=$5
                 AND dd.node_id = ANY (string_to_array($6, ',')::integer[])
                 AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
                 GROUP BY lower(dd.key1)
                 ORDER BY sum(value)/$1 DESC LIMIT 40)
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_cctld_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
             "SELECT alabel2ulabel(t.alabel) as x,
             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
             AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-')
             AND lower(d.key1) IN 
               (SELECT lower(dd.key1)
               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
                 LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
               WHERE nn.type_name='ccTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
               AND dd.starttime>=$4 AND dd.starttime<=$5 
               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
               GROUP BY lower(dd.key1) 
               ORDER BY sum(value)/$1 DESC LIMIT 40) 
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_newgtld = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
             "SELECT alabel2ulabel(t.alabel) as x,
             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t
             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
             AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
             AND lower(d.key1) IN 
               (SELECT lower(dd.key1) 
                 FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
                 LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
                 WHERE nn.type_name='New-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
                 AND dd.starttime>=$4 AND dd.starttime<=$5 
                 AND dd.node_id = ANY (string_to_array($6, ',')::integer[]) 
                 AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
                 GROUP BY lower(dd.key1) 
                 ORDER BY sum(value)/$1 DESC LIMIT 40) 
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_newgtld_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
             "SELECT alabel2ulabel(t.alabel) as x,
             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
             AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
             AND lower(d.key1) IN 
               (SELECT lower(dd.key1) 
               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
               WHERE nn.type_name='New-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
               AND dd.starttime>=$4 AND dd.starttime<=$5 
               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
               GROUP BY lower(dd.key1) 
               ORDER BY sum(value)/$1 DESC LIMIT 40) 
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_legacygtld = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
             "SELECT alabel2ulabel(t.alabel) as x, 
             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
             AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
             AND lower(d.key1) IN 
               (SELECT lower(dd.key1) 
               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = tt.alabel 
               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
               WHERE nn.type_name='Legacy-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
               AND dd.starttime>=$4 AND dd.starttime<=$5 
               AND dd.node_id = ANY (string_to_array($6, ',')::integer[]) 
               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
               GROUP BY lower(dd.key1) ORDER BY sum(value)/$1 DESC LIMIT 40) 
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_legacygtld_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
             "SELECT alabel2ulabel(t.alabel) as x,
             CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype', tlds t 
             where lower(d.key1) = lower(t.alabel) AND d.server_id=$2 AND d.plot_id=$3 
             AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
             AND lower(d.key1) IN 
               (SELECT lower(dd.key1) 
               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id 
               WHERE nn.type_name='Legacy-gTLD' AND dd.server_id=$2 AND dd.plot_id=$3 
               AND dd.starttime>=$4 AND dd.starttime<=$5 
               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
               GROUP BY lower(dd.key1) 
               ORDER BY sum(value)/$1 DESC LIMIT 40) 
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_othertld = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ, TEXT) AS",
             "SELECT alabel2ulabel(lower(d.key1)) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.node_id = ANY (string_to_array($6, ',')::integer[]) 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
             AND lower(d.key1) IN 
               (SELECT lower(dd.key1) 
               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = tt.alabel 
               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id WHERE nn.type_name is NULL 
               AND dd.server_id=$2 AND dd.plot_id=$3 
               AND dd.starttime>=$4 AND dd.starttime<=$5 
               AND dd.node_id = ANY (string_to_array($6, ',')::integer[]) 
               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
               GROUP BY lower(dd.key1) 
               ORDER BY sum(value)/$1 DESC LIMIT 40) 
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           qtype_vs_othertld_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, "(REAL, INTEGER, INTEGER, TIMESTAMPTZ, TIMESTAMPTZ) AS",
             "SELECT alabel2ulabel(lower(d.key1)) AS x, CASE WHEN il.name IS NULL THEN 'Other' ELSE il.name END AS key, sum(d.value)/$1 AS y 
             FROM dsc.data d LEFT OUTER JOIN iana_lookup il ON d.key2::text = il.value::text AND il.registry = 'qtype'
             WHERE d.server_id=$2 AND d.plot_id=$3 AND d.starttime>=$4 AND d.starttime<=$5 
             AND d.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
             AND lower(d.key1) IN 
               (SELECT lower(dd.key1) 
               FROM dsc.data dd LEFT OUTER JOIN tlds tt ON lower(dd.key1) = lower(tt.alabel) 
               LEFT OUTER JOIN tld_types nn ON tt.type=nn.id WHERE nn.type_name is NULL 
               AND dd.server_id=$2 AND dd.plot_id=$3 
               AND dd.starttime>=$4 AND dd.starttime<=$5 
               AND dd.key1 NOT IN ('-:SKIPPED_SUM:-', '-:SKIPPED:-') 
               GROUP BY lower(dd.key1) 
               ORDER BY sum(value)/$1 DESC LIMIT 40) 
            GROUP BY x, key;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},

           dnskey_vs_asn = {
             sql_joined <- paste("PREPARE", statementNm, select_nodes_prep,
               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
                 (SELECT key2::ipaddress AS xx, key1 AS key, sum(value)/$1 AS y
                  FROM dsc.data 
                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                  select_nodes_sql,
                  "AND key2", skipped_sql, "AND key1='48'",
                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},
           
           dnskey_vs_asn_all_nodes = {
             sql_joined <- paste("PREPARE", statementNm, all_nodes_prep,
               "SELECT ip2asn(xx) AS x, key, sum(y) as y FROM
                 (SELECT key2::ipaddress AS xx, key1 AS key, sum(value)/$1 AS y
                  FROM dsc.data 
                  WHERE server_id=$2 AND plot_id=$3 AND starttime>=$4 AND starttime<=$5",
                  "AND key2", skipped_sql, "AND key1='48'",
                  "GROUP BY xx, key ORDER BY y DESC LIMIT 40) AS sq
                group by x, key order by y desc;", sep=" ")
           sql=gsub("\n"," ",sql_joined)
           rs <- try(dbSendQuery(dsccon, sql))},


        )

        if(class(rs) == "try-error"){
            system(paste('logger -p user.crit database failed to load prepared statement name: "', statementNm, '" FROM dbconn.R', sep=""))
            return(FALSE)
        }else{
            return(TRUE)
        }
    }else{
        return(FALSE)
    }
}
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

#system('logger -p user.notice Hedgehog: Call to Hedgehog startup.R')

# Set the locale - For some reason this is set in R on the command line
# but not in Rapache!
Sys.setlocale("LC_ALL", 'en_US.UTF-8')

suppressPackageStartupMessages(library(Cairo))
suppressPackageStartupMessages(library(brew))
suppressPackageStartupMessages(library(DBI))
suppressPackageStartupMessages(library(RPostgreSQL))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(googleVis))
suppressPackageStartupMessages(library(reshape))
suppressPackageStartupMessages(library(RPostgreSQLHelper))
suppressPackageStartupMessages(library(digest))
suppressPackageStartupMessages(library(scales))
suppressPackageStartupMessages(library(yaml))
suppressPackageStartupMessages(library(plyr))
suppressPackageStartupMessages(library(grid))
suppressPackageStartupMessages(library(rjson))

dbdrv  <- dbDriver("PostgreSQL")
hh_config <- yaml.load_file("@HH_CONF_FILE@")
gui_config <- yaml.load_file(paste(hh_config$directories$conf, "/hedgehog_gui.yaml", sep=""))


# sink("/temp/R/R.out", append=TRUE)
# print(as.list(.GlobalEnv))
# sink()
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
# 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

handleTimeframe <- function() {
	
	if (hh_debug) {
		system('logger -p user.notice Hedgehog: In handleTimeFrame')
	}
    
	# Ensure data being pulled back is no more recent than presentation_delay_in_hours ago
	lag_end <- format(Sys.time() - (gui_config$www$presentation_delay_in_hours*60*60), '%Y-%m-%dT%H:%M')
	
	if (GET$stop > lag_end) {
		start_shift <- as.numeric(difftime(as.POSIXct(sub("T", " ", GET$stop)),as.POSIXct(sub("T", " ", GET$start)),units="hours"))
		GET$stop    <<- lag_end
		GET$start   <<- format(as.POSIXct(sub("T", " ", GET$stop)) - (start_shift*60*60), '%Y-%m-%dT%H:%M')
	}
}

# Check that all calls to this function have the required parameters
validateGetParams <- function() {
	
	if (hh_debug) {
		system('logger -p user.notice Hedgehog: In validateGetParams')
	}
	
	if (is.null(GET$type)) 
		return(FALSE)

    # Check that all calls to this function have the required parameters
	if (GET$type == "generatePlot" || GET$type == "generateYaml") {
		if (is.null(GET$start) || is.null(GET$stop)) {
			system('logger -p user.notice Hedgehog: Invalid time')
			return(FALSE)
		}
		if (is.null(GET$pltid)) 	 {
			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid pltid')
			return(FALSE)
		}
		if (is.null(GET$svrid))		 {
			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid svrid')
			return(FALSE)
		}
		if (is.null(GET$gvis))			 {
			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid gvis')
			return(FALSE)
		}
		if (is.null(GET$svrnm)) 				 {
			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid svrnm')
			return(FALSE)
		}
		if (is.null(GET$ndarr))					 {
			system('logger -p user.notice Hedgehog: In Hedgehog:validateGetParams - invalid ndarr')
			return(FALSE)
		}
		handleTimeframe()
	}   
	return(TRUE)
}
<!--
Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, you can obtain one at https://mozilla.org/MPL/2.0/.

Developed by Sinodun IT (www.sinodun.com)
-->

<!DOCTYPE html>
<!--[if lt IE 7 ]><html class="ie ie6" lang="en"> <![endif]-->
<!--[if IE 7 ]><html class="ie ie7" lang="en"> <![endif]-->
<!--[if IE 8 ]><html class="ie ie8" lang="en"> <![endif]-->
<!--[if (gte IE 9)|!(IE)]><!--><html lang="en"> <!--<![endif]-->
<head>

	<!-- Basic Page Needs
  ================================================== -->
	<meta charset="utf-8">
	<title>DNS-STATS: About Hedgehog</title>
	<meta name="Hedgehog" content="">
	<meta name="jad@sinodun.com" content="">

	<!-- Mobile Specific Metas
  ================================================== -->
	<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">

	<!-- CSS
  ================================================== -->
	<link rel="stylesheet" href="stylesheets/base.css?version=@HH_PACKAGE_VERSION@">
	<link rel="stylesheet" href="stylesheets/skeleton.css?version=@HH_PACKAGE_VERSION@">
	<link rel="stylesheet" href="stylesheets/layout.css?version=@HH_PACKAGE_VERSION@">

	<!--[if lt IE 9]>
		<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
	<![endif]-->

	<!-- Favicons
   ================================================== -->
	<link rel="shortcut icon" href="images/favicon.ico">
	<link rel="apple-touch-icon" href="images/apple-touch-icon.png">
	<link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
	<link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
        
</head>
    <body>
        <div class="container">
            <div class="sixteen columns">
                </br>
                <h2 class="remove-bottom">About DNS-STATS:Hedgehog</h2>
                <h5>Version @HH_PACKAGE_VERSION@: @HH_VERSION_NAME@ release</h5>
                <hr style='margin-bottom:10px;' />
                    <a href="hedgehog.html">DNS-STATS:Hedgehog</a> is a visualisation tool for DNS statistics that consumes data acquired with the 
                    <a href="http://dns.measurement-factory.com/tools/dsc/">DSC</a> collector. <br/> Hedgehog was initially developed for 
                    <a href="http://www.icann.org/">ICANN</a> by <a href="http://sinodun.com/">Sinodun IT</a> and is now released open 
                    source under the Mozilla Public License, version 2.0. For more information on the development and support of DNS-STATS:Hedgehog see 
                    the <a href="http://dns-stats.org/">dns-stats</a> website.<br>
                    The DNS-STATS:Hedgehog software releases are named after <a href="https://en.wikipedia.org/wiki/Hedgehog#Genera_and_species">subspecies</a> of hedgehogs.
                <hr />
                <ul>
                    <h5>Quick Help</h5></br>
                    <li><strong>Plots</strong>
                        <ul class="discs">
                            <li>The paradigm in Hedgehog is that the user selects a set of criteria (plot, time window, 
                                plot type and node selection) and then hits the 'Generate Plot!' button. (It is not the case
                                that changing e.g. the plot or node selection immediately re-generates the current plot.)</li>
                            <li>The static plots are .png files. Hedgehog offers the option to view these plots as normal
                                line plots, stacked line plots or line plots with a logarithmic y-axis.</li>
                            <li>The interactive plots are based on GoogleViz and provide zoom controls (fixed intervals 
                                at the top left and a slider bar at the bottom) and display data values along the top as the
                                user moves their mouse over the plot window.</li>
                            <li>Plots are cached as they are viewed. The 'day' plots (midnight-midnight) for the previous
                                day are cached overnight.</li>
                            <li>For line plots of less than or equal to one day interval the data points are 1 minute 
                                apart by default. Where the time interval is greater than 1 day the data is averaged to improve
                                the appearance of the plot. These averaged plots contain a fixed number of data points which 
                                results in the interval between the data points being 7 minutes for a 'week' plot and ~30 minutes
                                for a 'month' plot.</li>
                        </ul>
                    </li>
                    <li><strong>Time</strong>
                        <ul class="discs">
                            <li>The basic time controls generate graphs for fixed intervals. 
                                <ul class="circles">
                                    <li>The '&lt;' and '&gt;' buttons will shift the window forward or backwards by one interval unit.</li>
                                    <li>The '+&lt;' and '&gt;+' buttons will increase the window width by one interval unit. </li>
                                </ul>
                            </li>
                            <li>The advanced time controls allow the user to specify the time interval down to the minute.</li>
                        </ul>
                    </li>
                    <li><strong>Node selection</strong>
                        <ul class="discs">
                            <li>The nodes are displayed in different tabs for each top level Group, including an 'All' group.</li> 
                            <li>The nodes may be further grouped for display/selection purposes by 
                                either Country/City/Instance or not grouped at all. This grouping is controlled via the radio selection next to
                                the server drop down. This selection does not affect how nodes are grouped in plots; choose a different plot
                                under the 'Node Statistics' category to change the grouping in plots.</li>
                            <li>Within each Group tab, bulk node selections can be altered with the 'Actions' buttons.
                                The icons next to the Group/Country/City/Instance names indicate if all (full circle), some (half circle) or none (empty circle) of the nodes
                                in that choice are included in nodes for plot generation. Clicking on an individual Country/City/Instance/Node selects or de-selects that choice. </li>
                        </ul>
                    </li>
                    <li><strong>Static link</strong>
                         <ul class="discs">
                             <li>The static link provided below the plot window is a link to a cached version of the plot file on the Hedgehog server. 
                             This link may not be valid if the cache has been flushed since the plot was originally produced.</li>
                         </ul>
                     </li>
                </ul>
            </div>
            <div class="sixteen columns" id="copyright" >
                <hr />
                &copy; <a href="http://www.icann.org/">Internet Corporation for Assigned Names and Numbers</a> <br/>Developed by <a href="http://sinodun.com/">Sinodun Internet Technologies</a>.
            
              <hr style='margin-top:5px;' />
              This product includes GeoLite2 data created by MaxMind, available from
              <a href="http://www.maxmind.com">http://www.maxmind.com</a>.
            </div>
        </div>
    </body>
</html>
<%
# 
# Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, you can obtain one at https://mozilla.org/MPL/2.0/.

#
# Developed by Sinodun IT (www.sinodun.com)
#

# Debug mode set to 1, set to 0 to disable
hh_debug <<- 0

# TODO(refactor): preparing and executing DB queries should be wrapped better and in a dbQuery.R file
# TODO: use either tabs or spaces for indentation - we have both at the moment

# Initialise the static/interactive default type
initPltType <- function() {
	
	if (hh_debug) {
	    system('logger -p user.notice Hedgehog: In initPltType')
    }

    %><%=gui_config$www$default_plot_type%><%
}

# Initialise the Plot drop down menu contents
initPlotDDHtml <- function() {
	
	if (hh_debug) {
	    system('logger -p user.notice Hedgehog: In initPlotDDHtml')
    }

	# TODO: These queries should only return plots that are configured for display
	# Especially as not everyone will want the RSSAC plots showing
    if (!(prepStmnt("getpltddcategories", dsccon)) || !(prepStmnt("getpltid_ddname", dsccon))){
        %><option value="DB error">DB error preparing plot type query</option><%
        return(FALSE)
    }
   
    df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getpltddcategories;")
 
    if (is.null(df)) {
        %><option value="DB error">DB connection error</option><%
        return(FALSE)
    } 
	else if (nrow(df) < 1) {
    	%><option value="DB error">No plots found in DB</option><%
        return(FALSE)
    }
	
	for (i in 1:nrow(df)) {
		df2 <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getpltid_ddname('", df[i,1], "');", sep=""))
		if (is.null(df2)) {
			%><option value="DB error">DB connection error</option><%
			return(FALSE)
		} 
		else if (nrow(df2) < 1) {
			%><option value="DB error">No data found in DB</option><%
			return(FALSE)
		}
		optgroup_html <- sub(".*\\.","",df[i,1])
		%><optgroup label="<%=optgroup_html%>"><%
		for (i in 1:nrow(df2)) {
                if (! is.null(gui_config$www$enable_node_selection)) {
                        if ((gui_config$www$enable_node_selection == 0) && (df2[i,2] == "By node")) {
                                next
                        }
                } else {
                        system('logger -p user.warning Hedgehog: Missing enable_node_selection in config file. Using 1 enabled.')
                }
			%><option value="<%=df2[i,1]%>"><%=df2[i,2]%></option><%
		}
		%></optgroup><%
	}
    return(TRUE)
}

# Initialise the server drop down menu contents
initServerDDHtml <- function() {
	
	if (hh_debug) {
	    system('logger -p user.notice Hedgehog: In initServerDDHtml')
    }

    if (!(prepStmnt("getsrvrid_display_name", dsccon))) {
        %><option value="DB error">DB error preparing server query</option><%
        return(FALSE)
    }
    
    df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getsrvrid_display_name;")
    
    if (is.null(df)) {
        %><option value="DB error">DB connection error</option><%
        return(FALSE)
    } 
	else if (nrow(df) < 1) {
    	%><option value="DB error">No servers found in DB</option><%
        return(FALSE)
    }
    
    for (i in 1:nrow(df)) {
		%><option value="<%=df[i,1]%>"><%=df[i,2]%></option><%
	}
    return(TRUE)
}

getDefaultServer <- function() {
  
  if (hh_debug) {
      system('logger -p user.notice Hedgehog: In getDefaultServer')
  }
  
    # Calling code checks for no server specified
  %><%=gui_config$www$default_server%><%

  return(TRUE)
}

# Generate a nested JSON structure the js code can use to create the node tabs
initNodeData <- function() {
    if (hh_debug) {
        system('logger -p user.notice Hedgehog: In initNodeTabsHtml')
    }

    if (!(prepStmnt("getsrvrid_display_name", dsccon)) || !(prepStmnt("getgroups", dsccon)) || !(prepStmnt("getnodes", dsccon)) || !(prepStmnt("getnodesbyregion", dsccon))) {
        %>DB error preparing node query<%
        return(FALSE)
    }

    server=list()
    servers_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getsrvrid_display_name;")
    if (!is.null(servers_df) && nrow(servers_df) >= 1) {
        for (s in 1:nrow(servers_df)) {
            group_list=list()
            srvr_id <- servers_df[s,1]
            srvr_nm <- servers_df[s,2]
            groups_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getgroups(", srvr_id, ");", sep=""))
            
            for (g in 0:nrow(groups_df)) {
                nodes=list()
                if (g == 0) {
                    gp_nm = 'All'
                    nodes_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getnodes(", srvr_id, ");", sep=""))
                } else {
                    gp_nm <- groups_df[g,1]
                    # Other should really appear at the end of the list......
                    if  (gp_nm == 'Other'){
                        nodes_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getnodesbyregion(", srvr_id, ",'');", sep=""))
                    } else {
                        nodes_df <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getnodesbyregion(", srvr_id, ", '", gp_nm,"');", sep=""))
                    }
                }
                for (n in 1:nrow(nodes_df)) {
                    # Query fills in blank fields so we dont need to check here
                    nd_id      = nodes_df[n,1]
                    nd_nm      = nodes_df[n,2]
                    nd_country = nodes_df[n,3]
                    nd_city    = nodes_df[n,4]
                    nd_sg      = nodes_df[n,5]
                    node  = list(node_name=nd_nm, node_id=nd_id, node_country=nd_country, node_city=nd_city, node_sg=nd_sg)
                    nodes = c(nodes, list(node))
                }
                group=list(group_name=gp_nm, node_list=nodes)
                group_list=c(group_list, list(group))
            }
            server_list=list(server=srvr_nm, groups=group_list)
            server=c(server, list(server_list))
        }
        node_json = rjson::toJSON(server)
        output = cat(node_json)
        %><%output%><%
        return(TRUE)
    }
    %>No data returned<%
    return(FALSE)
}

getDefaultPlotId <- function() {
	
	if (hh_debug) {
	    system('logger -p user.notice Hedgehog: In getDefaultPlotId')
	}
	
	# Query the db to get the default plot id
	if (!(prepStmnt("getdefaultpltid", dsccon))) {
		return(FALSE)
	}
	
	defaultpltnm <- gui_config$www$default_plot_name
	if (is.null(defaultpltnm)) {
		system('logger -p user.warning Hedgehog: Missing Default Plot name in config file. Using By region.')
		defaultpltnm <- "by_region"
	}
	defaultpltid <- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getdefaultpltid('", defaultpltnm, "');", sep=""))	
	
	if (is.null(defaultpltid) || nrow(defaultpltid) == 0) {
		return(FALSE)
	}
	%><%=as.integer(defaultpltid[1])%><%
}

getEnableNodeSelection <- function() {
        if (hh_debug) {
            system('logger -p user.notice Hedgehog: In getEnableNodeSelection')
        }
        ens <- gui_config$www$enable_node_selection
        if (is.null(ens)) {
                system('logger -p user.warning Hedgehog: Missing enable_node_selection in config file. Using 1 enabled.')
                ens <- 1
        }
        %><%=ens%><%
}

getDefaultGrouping <- function() {
	
	if (hh_debug) {
	    system('logger -p user.notice Hedgehog: In getDefaultGrouping')
	}
	
	grouping = gui_config$www$default_node_grouping
	if (is.null(grouping)) {
		system('logger -p user.warning Hedgehog: Missing Node Grouping in config file. Using None.')
		%>none<%
		return(TRUE)
	}

	if      (grouping == "instance") {%>instance<%} 
	else if (grouping == "city")     {%>city<%}
	else if (grouping == "country")  {%>country<%} 
	else if (grouping == "none")     {%>none<%}
	else {
		system('logger -p user.warning Hedgehog: Unsupported Node Grouping present in config file. Using Instance.')
		%>instance<%
	}
	return(TRUE)
}

getSupportURL <- function() {
	
	if (hh_debug) {
	    system('logger -p user.notice Hedgehog: In getSupportURL')
	}

    # Calling code checks for missing url
	%><%=gui_config$www$support_url%><%

	return(TRUE)
}

generatePlot <- function(){

    #TODO(refactor): Move more of this code out to the generatePlot.R file

	if(hh_debug){
		system('logger -p user.notice Hedgehog: In generatePlot')
	}

	source(paste(hh_config$directories$web_source, "/R/generatePlot.R", sep=""))
    
	# Query the db to get the plot attributes
	if (!(prepStmnt("getpltdetails", dsccon))) {
		return(FALSE)
	}

	dd_pltid			<- strtoi(GET$pltid)
	pltdetails			<- dbGetDataFrame(dbdrv, dsccon, dbconstr, paste("EXECUTE getpltdetails(", dd_pltid, ");", sep=""))	
	
	if (is.null(pltdetails)) {
		return(FALSE)
	}
	
	pltnm				<- pltdetails$name
	plttitle			<- pltdetails$title
	pltdescription		<- pltdetails$description
	pltid				<- pltdetails$plot_id
		
	svrid				<- GET$svrid
	gvis				<- GET$gvis
	ndarr				<- GET$ndarr
	svrnm				<- GET$svrnm
	start				<- GET$start
	stop				<- GET$stop
	
	# Check the parameters
	if (!(prepStmnt("getsrvrid_display_name", dsccon))) {
		return(FALSE)
	}
	serverlist <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getsrvrid_display_name;")
	if (is.null(serverlist)) {
		return(FALSE)
	}
	check <- grep(svrnm,serverlist$display_name)
	if (length(check) == 0) {
		return(FALSE)
	}
	sid <- serverlist$id[check]
	if (sid != svrid){
		return(FALSE)
	}
		
	# Initialise plot parameters
	initPlotOptions()
	
	
	# Initialise name of plot file and make sure the cache directory exists
	plot_dir = paste(hh_config$directories$plotcache, "/", svrnm, "/", pltnm, sep="") 
	dir.create(file.path(plot_dir), showWarnings = FALSE, recursive = TRUE)
	if (pltnm %in% unknown_graphs ) {
		plot_file = "plots/no_graph.png"
	} 
	else {
		plot_file = paste(plot_dir, "/", start, "-", stop, "-all.png", sep="")
		if (gvis == 1) {
			plot_file <- sub(".png", ".html", plot_file)
		}
		if (ndarr != '-1') {
			plot_file <- sub("-all.", paste("-", digest(ndarr, algo="sha256"), ".", sep=""), plot_file)
		}
	}

	# convert the times to POSIX string
	simple_start   <- as.POSIXct(start, format='%Y-%m-%dT%H:%M')
	simple_stop    <- as.POSIXct(stop, format='%Y-%m-%dT%H:%M')

	cache       <- FALSE # Create new plot_file
	if (file.exists(plot_file) && gui_config$www$use_plot_caching == 1) {
		lag_hours <- as.numeric(difftime(file.info(plot_file)$mtime,as.POSIXct(simple_start),units="hours"))
		if (lag_hours > gui_config$www$caching_delay_in_hours) {
			cache <- TRUE # Pull back file from disk
		}
	}

	if (!cache) {	
		plot_file <- generatePlotFile(plttitle, pltnm, dd_pltid, plot_file, simple_start, simple_stop, svrid, pltid, gvis, ndarr, dsccon)
	}
    
	# OK - we have a plot file so now lets generate the html to hold that
	now <- format(Sys.time(), '%Y-%m-%dT%H:%M')
	
	plot_file_raw = gsub('../','/',plot_file)
	if (plot_file != "plots/no_graph.png" && plot_file != "plots/no_results.png" && plot_file != "plots/no_connection.png") {
		plot_file <- sub(hh_config$directories$plotcache, "../plotcache", plot_file)
        
		if (gvis == '1') {
			%><div id="plot">
				<hr />
				<iframe scrolling="no" width="100%" height="600px" frameborder="0" id="plotframe" src="<%=plot_file%>?nocache=<%=now%>"></iframe>
			</div><%
		}
		else if (pltnm %in% log_option) {
			log_file <- sub(".png", "-log.png", plot_file)
			stack_file <- sub(".png", "-stack.png", plot_file)
			%><div id="opts" class="" style="height=10px">
				<hr style="margin-bottom:10px;"/>
				<input type="radio" name="pltoptions" id="rbliny" onclick="toggle_plot()" title="Line plot with linear y-axis" checked>
				<label for="rbliny" title="Line plot with linear y-axis">Linear scale</label>
				<input type="radio" name="pltoptions" id="rblogy" onclick="toggle_plot()" title="Line plot with log y-axis">
				<label for="rblogy" title="Line plot with log y-axis">Log scale</label>
				<input type="radio" name="pltoptions" id="rbstack" onclick="toggle_plot()" title="Stacked line plot with linear y-axis">
				<label for="rbstack"  title="Stacked line plot with linear y-axis">Stacked linear</label>
			</div>
			<div id="plot">
				<img src="<%=plot_file%>?nocache=<%=now%>" id= "liny" name="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',plot_file)%>"/>
				<img src="<%=log_file%>?nocache=<%=now%>" class="hidden" id="logy" name="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',log_file)%>" />
				<img src="<%=stack_file%>?nocache=<%=now%>" class="hidden" id="stack" name="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',stack_file)%>"/>
			</div><%
		}
		else {
			%><div id="plot">
				<hr />
				<img id="staticplot" src="<%=plot_file%>?nocache=<%=now%>" />
			</div><%
		}
		%><div id="plink" class="" style="height:27px; width:960px;">
			<label style="font-size:80%" for="ploturi">Static Link</label>
			<input type="text" id="ploturi" readonly="readonly" value="http://<%=SERVER$hostname%><%=gsub("\\.\\./",'/',plot_file)%>" style="width: 870px; height:7px; font-size:80%">
		</div><%
	}
	else {
		%><div id="plot">
			<hr />
			<img src="<%=plot_file%>" />
		</div><%
	}
}

validateDbVersion <- function() {
	
	if (hh_debug) {
		system('logger -p user.notice Hedgehog: In validateDbVersion')
	}

	if (!(prepStmnt("getdbversion", dsccon))) {
		%>Error: preparing db query<%
		return(FALSE)
	}
		
	dbversion <- dbGetDataFrame(dbdrv, dsccon, dbconstr, "EXECUTE getdbversion;")
    
	if (is.null(dbversion)) {
		%>Error: DB connection error<%
		return(FALSE)
	}
	
	if (dbversion != DB_SCHEMA_VERSION) {
		%>Error: Database version mismatch. Contact an administrator.<%
		return(FALSE)
	}	

	return(TRUE)
}

#########################################################
###########  Entry point for brew calls #################
#########################################################

if (hh_debug) {
    system('logger -p user.notice Hedgehog: Call to Hedgehog brew')
}

source(paste(hh_config$directories$web_source, "/R/utilities.R", sep = ""))
if (!validateGetParams()) {
	%>Error: invalid parameters in method call.<%
	return()
}

if (GET$type == "initPltType") {
	# this doesn't need to call the DB
	initPltType()
	return()
}

if (is.null(hh_config$database$read_pass)) { 
    db_password <- ""
} else {
    db_password <- hh_config$database$read_pass
}
dsccon <- try(dbConnect(dbdrv, dbname = hh_config$database$name, user = hh_config$database$read_user, port = hh_config$database$port, host = hh_config$database$host, password = db_password))
dbconstr <<-paste('dbname = \'', hh_config$database$name, '\', user = \'', hh_config$database$read_user, '\', port = ', hh_config$database$port, ', host = \'', hh_config$database$host, '\', password = \'', hh_config$database$read_pass,'\'', sep="")
if (hh_debug) {
  msg <- paste('logger -p user.notice dsccon: ', dbconstr)
  system(msg)
}

if (class(dsccon) != "try-error") {
	dbSendQuery(dsccon, "set statement_timeout to 300000;")
	source(paste(hh_config$directories$web_source, "/R/prepareStatement.R", sep = ""))
	source(paste(hh_config$directories$web_source, "/R/generatePlot.R", sep = ""))
	switch(GET$type,
		validateDBVersion  = {validateDbVersion()},
		initPlotDDHtml     = {initPlotDDHtml()},
		initServerDDHtml   = {initServerDDHtml()},
		getDefaultServer   = {getDefaultServer()},
		initNodeTabsHtml   = {initNodeTabsHtml()},
		initNodeData       = {initNodeData()},
		getDefaultPlotId   = {getDefaultPlotId()},
		getDefaultGrouping = {getDefaultGrouping()},
		getSupportURL      = {getSupportURL()},
		generatePlot       = {generatePlot()},
		generateYaml       = {generateYaml(dsccon)},
		getEnableNodeSelection = {getEnableNodeSelection()}
	)
	dbDisconnect(dsccon)
} else {
	%>Error: Could not connect to the database - please check the connection parameters: name, read-user, port<%
	return(FALSE)
}
%>
<!--
Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, you can obtain one at https://mozilla.org/MPL/2.0/.

Developed by Sinodun IT (www.sinodun.com)
-->

<!DOCTYPE html>
<!--[if lt IE 7 ]><html class="ie ie6" lang="en"> <![endif]-->
<!--[if IE 7 ]><html class="ie ie7" lang="en"> <![endif]-->
<!--[if IE 8 ]><html class="ie ie8" lang="en"> <![endif]-->
<!--[if (gte IE 9)|!(IE)]><!--><html lang="en"> <!--<![endif]-->
<head>

	<!-- Basic Page Needs
  ================================================== -->
	<meta charset="utf-8">
	<title>DNS-STATS:Hedgehog</title>
	<meta name="Hedgehog" content="">
	<meta name="jad@sinodun.com" content="">

	<!-- Mobile Specific Metas
  ================================================== -->
	<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">

	<!-- CSS
  ================================================== -->
	<link rel="stylesheet" href="stylesheets/base.css?version=@HH_PACKAGE_VERSION@">
	<link rel="stylesheet" href="stylesheets/skeleton.css?version=@HH_PACKAGE_VERSION@">
	<link rel="stylesheet" href="stylesheets/layout.css?version=@HH_PACKAGE_VERSION@">

	<!--[if lt IE 9]>
		<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
	<![endif]-->

	<!-- Favicons
   ================================================== -->
	<link rel="shortcut icon" href="images/favicon.ico">
	<link rel="apple-touch-icon" href="images/apple-touch-icon.png">
	<link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
	<link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
        
</head>
<body>
  <div class="container">
    <hr style='margin-bottom:5px; margin-top:1px;'/>
    <div class="three columns">
      <div class="logo" id="logo">
          <h5 class="remove-bottom" style="text-align: center; font-weight: bold">DNS-STATS</h5>
          <h5 class="remove-bottom" style="text-align: center">Hedgehog @HH_PACKAGE_VERSION@:<br>
          <a href="https://en.wikipedia.org/wiki/Hedgehog#Genera_and_species">@HH_VERSION_NAME@ release</a></h5>
      </div>
    </div>
    <form method="get" name="plotParams">
      <div class="four columns" id="timedisplay"><!-- 
        auto populated by datetime.js
   --></div>
      <div class="five columns" id="timeTabs">
        <div class="basicadv">
          <a id="tfbasic" class="strong">Basic Time</a> | <a id="tfadv">Advanced time</a> <input type="button" id="setToday" value="Today" title="Set 'To' date equal to now and 'From' date equal to this time yesterday"> 
        </div>
        <div id="timeTabsBasic">
          <div class="mycolumn" id="lefttimebtns"><!--
         --><input type="button" id="lextend" value="+<" title="Extend time frame window by selected unit" ><!--
         --><input type="button" id="larrow" value="<" title="Shift time frame window" ><!--
       --></div>
          <div class="mycolumn" id="tframebtns"><!--
         --><input id="1h" type="radio" name="tframe" value="1"><!--
         --><label class="button" for="1h">1 hr</label><!--
         --><input id="4h" type="radio" name="tframe" value="4"><!--
         --><label class="button" for="4h">4 hrs</label><!--
         --><input id="12h" type="radio" name="tframe" value="12"><!--
         --><label class="button" for="12h">12 hrs</label><!--
         --><br /><!--
         --><input id="day" type="radio" name="tframe" value="24"><!--
         --><label class="button" for="day">day</label><!--
         --><input id="wk" type="radio" name="tframe" value="168"><!--
         --><label class="button" for="wk">week</label><!--
         --><input id="mnth" type="radio" name="tframe" value="730"><!--
         --><label class="button" for="mnth">month</label><!--
       --></div>
          <div class="mycolumn" id="righttimebtns"><!--
         --><input type="button" id="rarrow" value=">" title="Shift time frame window" ><!--
         --><input type="button" id="rextend" value=">+" title="Extend time frame window by selected unit" ><!--
       --></div>
        </div>
        <div id="timeTabsAdv">
          <div>
            <label for="start" >Start</label>
            <input type="datetime-local" name="start" id="start" />
          </div>
          <div>
            <label for="stop" >End</label>
            <input type="datetime-local" name="stop" id="stop" />
          </div>
        </div>
      </div>
      <div class="four columns" id="plotselection">
        <div id="radioplt">
            <input type="radio" name="rbplot" id="ggplot" title="Static PNG plot" checked>
            <label for="ggplot" title="Static PNG plot">Static plot</label>
            <input type="radio" name="rbplot" id="googleviz" title="Live Google Viz plot">
            <label for="googleviz" title="Live Google Viz plot">Interactive plot</label>
        </div>
        <label for="plotType">Plot</label>
        <select name="plotType" id="plotType"><!-- 
        auto populated by gengraph.js
   --></select>
        <input id="generate" name="gobutton" type="button" value="Generate Plot!">  
      </div>
      <div class="sixteen columns" id="userdisplay"><!-- 
        auto populated by datetime.js
   --></div>   
      <div class="sixteen columns" id="outerplot"><!-- 
        auto populated by gengraph.js
   --></div>
      <hr style='margin-top:5px;margin-bottom:5px'/>
      <div class="sixteen columms" id="srvr_select">
      <label for="servers">Servers</label>
      <select name="servers" id="servers" onchange="serverTab()">
<!-- 
        auto populated by gengraph.js
   --></select>
      </div>
      <div class="sixteen columns" id="nodetabs"><!-- 
         auto populated by nodetabs.js
   --></div>
    </form>
    <div class="eight columns" id="local_links" >
      <hr style='margin-bottom:5px;' />
      Local Links: &nbsp; <a href="../rssac" target="_blank">RSSAC reports</a> &nbsp; <a href="about.html" target="_blank">Hedgehog QuickHelp</a>
    </div>
    <!-- TODO: Better job of links -->
    <div class="eight columns" id="ext_links" >
      <hr style='margin-bottom:5px;' />
      External Links:  &nbsp; <a href="https://github.com/dns-stats/hedgehog/wiki" target="_blank">Hedgehog Wiki</a> &nbsp; <a href="http://dns-stats.org/" target="_blank">dns-stats.org</a>
      &nbsp; <a id="support_link" href="https://github.com/dns-stats/hedgehog/issues" target="_blank">Support</a>
    </div>
    <div class="sixteen columns" id="copyright" >
      <hr style='margin-top:5px;' />
      &copy; <a href="http://www.icann.org/">Internet Corporation for Assigned Names and Numbers</a> <br/>Developed by <a href="http://sinodun.com/">Sinodun Internet Technologies</a>.
    
      <hr style='margin-top:5px;' />
      This product includes GeoLite2 data created by MaxMind, available from
      <a href="http://www.maxmind.com">http://www.maxmind.com</a>.
	  <br/>Powered by: <a href="http://www.postgresql.org">PostgreSQL</a>, <a href="http://rapache.net/index.html">RApache</a>, <a href="https://httpd.apache.org">Apache</a>, 
	  <a href="https://developers.google.com/chart/?hl=en">Google Charts</a>, <a href="https://www.r-project.org">R</a>, <a href="https://jquery.com">jQuery</a>, <a href="https://getdnsapi.net">getdns</a>
    </div>  
  </div><!-- container -->

</body>

    <!-- Javascript
  ================================================== 
    <script language="javascript" type="text/javascript"
             src="/javascript/jquery/jquery.min.js"></script>
    <script language="javascript" type="text/javascript"
             src="/javascript/jquery-ui/jquery-ui.min.js"></script> -->
    <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
    <script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js"></script>
    <script src="js/gengraph.js?version=@HH_PACKAGE_VERSION@" type="text/javascript"></script>
    <script src="js/datetimes.js?version=@HH_PACKAGE_VERSION@" type="text/javascript"></script>
    <script src="js/nodetabs.js?version=@HH_PACKAGE_VERSION@" type="text/javascript"></script>

</html>
///
/// Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
///
/// This Source Code Form is subject to the terms of the Mozilla Public
/// License, v. 2.0. If a copy of the MPL was not distributed with this
/// file, you can obtain one at https://mozilla.org/MPL/2.0/.
///
/// Developed by Sinodun IT (www.sinodun.com)
///

$(document).ready(function() {
    // This function is called when hedgehog.html has loaded.
    // Any other document.ready functions within included javascript
    // files will also be called non-deterministically.
    
    // register callback function for when the advanced time input selection link is clicked
    $("#tfadv").click(function() {
        $("#timeTabsBasic").addClass("hidden");
        $("#tfbasic").removeClass("strong");
        $("#timeTabsAdv").removeClass("hidden");
        $("#tfadv").addClass("strong");
    });
    
    // register callback function for when the basic time input selection link is clicked
    $("#tfbasic").click(function() {
        $("#timeTabsAdv").addClass("hidden");
        $("#tfadv").removeClass("strong");
        $("#timeTabsBasic").removeClass("hidden");
        $("#tfbasic").addClass("strong");
    });
    
    // this function is no longer used, since we removed the debug option from the production release
    $("#debug").click(function() {
        $("#debugadv").toggleClass("strong");
        $("#debug").toggleClass("strong");
    });
    
    // register a callback function for when the Today button is clicked
    $("#setToday").click(function set_today() {
        window.stop = new Date();
        window.stop.setUTCSeconds(0,0);
        set_start_to_stop_dt();
        set_start_to_stop_tm();
        window.start.setUTCMinutes(window.start.getUTCMinutes() + 1);
        getNewDateTime(window.start, -24);
        setAdvInputs();
        setUserDisplayMsg();
        $('#day').prop('checked', true);
    });
    
    // register a callback function for when the basic time control +< button is clicked
    $("#lextend").click(function() {
        extend_tframe("left");
    });
    
    // register a callback function for when the basic time control >+ button is clicked
    $("#rextend").click(function() {
        extend_tframe("right");
    });
    
    // register a callback function for when the basic time control < button is clicked
    $("#larrow").click(function() {
        shift_tframe("left");
    });
    
    // register a callback function for when the basic time control > button is clicked
    $("#rarrow").click(function() {
        shift_tframe("right");
    });
    
    // register a callback function for when any basic time control time frame buttons are clicked
    $("input:radio[name='tframe']").change(function() {
       set_tframe(parseInt($("input:radio[name='tframe']:checked").val()));
    });
    
    // register a callback function for when the advanced time control start input is changed
    $("#start").change(function() {
        new_start_str = $('#start').val() + ":00Z"
        if (window.start !== new_start_str) {
            window.start = new Date(new_start_str);
            setUserDisplayMsg();
        }            
    });
    
    // register a callback function for when the advanced time control stop input is changed
    $("#stop").change(function() {
        new_stop_str = $('#stop').val() + ":00Z"
        if (window.stop !== new_stop_str) {
            window.stop = new Date(new_stop_str);
            setUserDisplayMsg();
        }
    });
    
});


function setUserDisplayMsg() {
    // update timedisplay and userdisplay messages
    var tmsg = '<p class="remove-bottom"><span class="strong">From </span>' + window.start.toUTCString().replace('GMT','UTC') + '<br /><span class="strong">To </span>' + window.stop.toUTCString().replace('GMT','UTC') + '</p>';
    $('#timedisplay').html(tmsg);
    
    enableGenerate(true);
    
    var existing = $('#userdisplay').html();
    var invalidmsg = '<p class="error remove-bottom"><span class="strong">Error:</span> you have selected an End Date earlier than or equal to the Start date - please make a new selection</p>'; 
    
    if(window.stop <= window.start) {
        enableGenerate(false);
        if (existing.indexOf(invalidmsg) === -1) $('#userdisplay').html(existing + invalidmsg);
    } else {
        $('#userdisplay').html(existing.replace(invalidmsg, ""));
    }
    
    var existing = $('#userdisplay').html();
    var futuremsg = '<p class="warning remove-bottom"><span class="strong">Warning:</span> you have selected a future End Date - by default you will only see data up to the current moment</p>';
    
    if((window.stop > new Date()) || (window.start > new Date())){
        if (existing.indexOf(futuremsg) === -1) $('#userdisplay').html(existing + futuremsg);
    } else {
        $('#userdisplay').html(existing.replace(futuremsg, ""));
    }
}

function getNewDateTime(d, hours){
    // add / subtract the number of hours from datetime, d,
    // according to switch below and return new datetime
    // default will return the same date (no default case)
    switch(hours) {
        case -730:
            d.setMonth(d.getMonth() - 1); //d - 1 month
            break;
        case -168:
            d.setDate(d.getDate() - 7); //d - 1 week
            break;
        case -24:
            d.setDate(d.getDate() - 1); //d - 1 day
            break;
        case -12:
            d.setHours(d.getHours() - 12); //d - 12 hours
            break;
        case -4:
            d.setHours(d.getHours() - 4); //d - 4 hours
            break;
        case -1:
            d.setHours(d.getHours() - 1); //d - 1 hour
            break;
        case 1:
            d.setHours(d.getHours() + 1); //d + 1 hour
            break;
        case 4:
            d.setHours(d.getHours() + 4); //d + 4 hours
            break;
        case 12:
            d.setHours(d.getHours() + 12); //d + 12 hours
            break;
        case 24:
            d.setDate(d.getDate() + 1); //d + 1 day
            break;
        case 168:
            d.setDate(d.getDate() + 7); //d + 1 week
            break;
        case 730:
            d.setMonth(d.getMonth() + 1); //d + 1 month
            break;
    }
}

function reset_hours() {
    window.start.setUTCHours(0,0,0);
    window.stop.setUTCHours(23,59,0);
}

function set_start_to_stop_dt() {
    // sets start date equal to stop date
    window.start.setUTCFullYear(window.stop.getUTCFullYear(), window.stop.getUTCMonth(), window.stop.getUTCDate());
}

function set_start_to_stop_tm() {
    // sets start time equal to stop time
    window.start.setUTCHours(window.stop.getUTCHours(), window.stop.getUTCMinutes(), 0, 0);
}

function set_tframe(hours) {
    set_start_to_stop_dt();
    set_start_to_stop_tm();
    switch(hours) {
        case 1:
        case 4:
        case 12:
            window.start.setUTCHours(window.start.getUTCHours() - hours);
            break;
        case 24:
            window.start.setUTCDate(window.start.getUTCDate() - 1);          
            break;
        case 168:
            window.start.setUTCDate(window.start.getUTCDate() - 7);
            break;            
        case 730:
            window.start.setUTCMonth(window.start.getUTCMonth() - 1);
            break;
        default: // do a day
            window.start.setUTCDate(window.start.getUTCDate() + 1);          
            break;
    }
    window.start.setUTCMinutes(window.start.getUTCMinutes() + 1);
    setAdvInputs();
    setUserDisplayMsg();    
}

function shift_tframe(direction) {
    var hours = $("input[type='radio'][name='tframe']:checked").val() * 1; // current time frame increment selected
    if (direction === 'left') {
        hours = -hours;
    }
    getNewDateTime(window.start, hours);
    getNewDateTime(window.stop, hours);
    setAdvInputs();
    setUserDisplayMsg();
}

function extend_tframe(direction) {
    var hours = $("input[type='radio'][name='tframe']:checked").val() * 1; // current time frame increment selected
    if (direction === 'left') {
        getNewDateTime(window.start, -hours);
    }else if (direction === 'right') {
        getNewDateTime(window.stop, hours);
    }
    setAdvInputs();
    setUserDisplayMsg(); 
}

function setAdvInputs() {
    // sets the timeTabsAdv inputs
    // slice removes the final ":###Z" which is not accepted by datetime input
    $('#start').val(window.start.toISOString().slice(0,16));
    $('#stop').val(window.stop.toISOString().slice(0,16));
}
///
/// Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
///
/// This Source Code Form is subject to the terms of the Mozilla Public
/// License, v. 2.0. If a copy of the MPL was not distributed with this
/// file, you can obtain one at https://mozilla.org/MPL/2.0/.
///
/// Developed by Sinodun IT (www.sinodun.com)
///

$(document).ready(function() {
    // This function is called when hedgehog.html has loaded.
    // Any other document.ready functions within included javascript
    // files will also be called non-deterministically.
    
    // hide advanced time controls
    $("#timeTabsAdv").addClass("hidden");

    // pre-select day from basic time handling
    $('#day').prop('checked', true);
    
    // initialise start and stop datetimes
    window.start = new Date();
    window.start.setUTCDate(window.start.getUTCDate() -1);
    window.stop = new Date(window.start);
    reset_hours();
    setAdvInputs();
    setUserDisplayMsg();
    
    // initialise customMsg
    window.customMsg = null;
    
    // initialise lastPlot and selectedOpt
    window.lastPlot = null;
    window.selectedOpt = null;

    // populate plot and server dropdowns and nodetabs then generate default plot    
    // , brew("initnodeslist"),
    $.when(brew("validateDBVersion"),brew("initPltType"), brew("initPlotDDHtml"), brew("initServerDDHtml"), 
           brew("getDefaultServer"), brew("initNodeData"), brew("getDefaultGrouping"), brew("getDefaultPlotId"), 
           brew("getSupportURL"), brew("getEnableNodeSelection")).done(function(db,rp, pt, ss, ds, nd, dg, dp, su, ns){

        if(db[0].indexOf("Error: Database version mismatch.") > -1) {
            setDbVersionlMsg(true);
            $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_connection.png" /><hr /></div>');
            return;
        }else if(db[0].indexOf("Error: Could not connect to the database") > -1) {
            setDbFailMsg(true);
            $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_connection.png" /><hr /></div>');
            return;
        }

        //initialise plot radio selection
        if (rp[0].indexOf('interactive') !== -1) {
            $('#googleviz').prop('checked', true);
        }

        // initialise plot drop down
        $("#plotType").html(pt[0]);
        $("#plotType").val(parseInt(dp[0]));

        // Initialise Server drop down item
        $("#servers").html(ss[0]);

        // check we have at least 1 server
        var servers_ok = server_list_has_content();
        if(!servers_ok) {
            alert("Error: No servers are configured in the database");
            $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_results.png" /><hr /></div>');
            setNoResultsMsg(true);
            return;
        }

        // checks user defined default for undefined, null and empty string
        if (!(!ds[0] || ds[0].length === 0 || !ds[0].trim())) {
            var options= document.getElementById('servers').options;
            for (var i = 0; i < options.length; i++) {
                if (options[i].text === ds[0].trim()) {
                    options[i].selected = true;
                    break;
                }
            }
        }

        // initialise node tabs and grouping
        initNodeHtml(nd[0], ns[0]);
        $('#ng_' + dg[0]).prop('checked', true);

        setServersGroups();
        initnodetabs();
        serverTab();
        
        // checks for undefined, null and empty string
        if (!(!su[0] || su[0].length === 0 || !su[0].trim())) {
            $("#support_link").attr('href', su[0]);
        }

        // generate default plot
        genDSCGraph();
        setInitHelpMsg(true);

    });
    
    // register callback function for when the static / interactive radio buttons are clicked
    $("input[type='radio'][name='rbplot']").click(function() {
        enableGenerate(true);
    });
    
    // register callback function for when the plot drop down selection is changed
    $("#plotType").change(function() {
       enableGenerate(true);
    });

    // register callback function for when the generate plot button is clicked
    $("#generate").click(function() {
        genDSCGraph();
    });

});

function enableGenerate(ind) {
    // enable or disable the generate plot button
    if(ind){
        $('#generate').prop("disabled", false);
        $('#generate').attr("title", "Select all the criteria for your plot, then click 'Generate Plot'");
        setNoResultsMsg(false);
        setInitHelpMsg(false);
    }else{
        $('#generate').prop("disabled", true);
        $('#generate').attr("title", "Select new plot criteria to activate this button.");
    }
}

function brew(divId) {
    // call hedgehog.brew with type parameter set to the calling divId
    return $.ajax({
        url: "brew/hedgehog.brew",
        data: { 'type': divId}
    });
}

function genDSCGraph() {
    // call hedgehog.brew to generate a plot
    var gvis = 0;
    if ($('#googleviz').prop('checked') === true) gvis = 1;

    var pltid = $("#plotType option:selected").val();
    var svrnm_tmp = $("#servers option:selected").text();
    var svrnm = parse_name(svrnm_tmp);
    var svrid = $("#servers option:selected").val();
        
    var ndset = {};
    var allselected = true;
    $("input[type='checkbox'].nodeselection").each(function(){
        if( $(this).attr('id').split('_')[2] === svrnm) {
            if( $(this).is(':checked') ) {
                var node_id = $(this).attr('id').split("_")[0];
                ndset[node_id] = true;
            }else{
                allselected = false;
            }
        }
    });
    var ndarr = "";
    if (allselected){
        ndarr = -1;
    }else{
        for(var node in ndset){
            ndarr = ndarr + node + ",";
        }
        ndarr = ndarr.substring(0, ndarr.length - 1);
        if(ndarr === '') {
            window.alert("You must select at least one node")
            return
        }
    }
    
    var request = $.ajax({
        url: "brew/hedgehog.brew",
        data: { 'type': "generatePlot",
                'start': window.start.toISOString().slice(0,16),
                'stop': window.stop.toISOString().slice(0,16),
                'gvis': gvis,
                'pltid': pltid,
                'svrnm' : svrnm,
                'svrid': svrid,
                'ndarr': ndarr},
        beforeSend: function() {
                enableGenerate(false);
                $('body').css('cursor', 'wait');
                $("#outerplot").html('<hr /><img src="images/ajax-loader.gif" alt="loading..." /><hr />');
        }
    });
            
    request.done(function(data) {
        $("#outerplot").html(data);
        $('body').css('cursor', 'auto');
        setUserMsg(null);
        if (window.lastPlot === pltid){
            if(window.selectedOpt !== null) {
                if ( $(window.selectedOpt).prop('checked') === false ){
                    $(window.selectedOpt).prop('checked', true);
                    toggle_plot();
                }
                
            }
        }
        window.lastPlot = pltid;
    });
    
    request.fail(function(jqXHR, textStatus) {
        $("#outerplot").html('<div class="sixteen columns" id="plot"><hr /><img src="plots/no_results.png" /><hr /></div>');
        $('body').css('cursor', 'auto');
        enableGenerate(true);
        setUserMsg("Request failed with status '" + textStatus + "'");
        window.lastPlot = null;
    });
}


function toggle_plot() {  
    // hide and display linear, stacked or log plots based on user selection
    if ($("#rbliny").prop('checked')) {
        window.selectedOpt = "#rbliny";
        $('#liny').removeClass('hidden');
        $('#logy').addClass('hidden');
        $('#stack').addClass('hidden');
        $("#ploturi").val($("#liny").prop("name"));
    } else if ($("#rblogy").prop('checked')) {
        window.selectedOpt = "#rblogy";
        $('#liny').addClass('hidden');
        $('#logy').removeClass('hidden');
        $('#stack').addClass('hidden');
        $("#ploturi").val($("#logy").prop("name"));
    } else if ($("#rbstack").prop('checked')) {
        window.selectedOpt = "#rbstack";
        $('#liny').addClass('hidden');
        $('#logy').addClass('hidden');
        $('#stack').removeClass('hidden');
        $("#ploturi").val($("#stack").prop("name"));
    }
}

function setCustomMsg(msg) {
    var existing = $('#userdisplay').html();
    if (window.customMsg !== null) {
        if (existing.indexOf(window.customMsg) !== -1) $('#userdisplay').html(existing.replace(window.customMsg, ""));
        window.customMsg = null;
    }
    if (msg !== null) {
        window.customMsg = '<p class="error remove-bottom"><span class="strong">Error:</span> ' + msg + '</p>';
        if (existing.indexOf(window.customMsg) === -1) $('#userdisplay').html(existing + window.customMsg);
    }
}

function setInitHelpMsg(ind) {
    var existing = $('#userdisplay').html();
    var msg = '<p class="info remove-bottom"><span class="strong">Info:</span> To generate a new plot, change the time, node or plot type and click \'Generate Plot.\'</p>';
    if (ind) {
        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
    } else {
        $('#userdisplay').html(existing.replace(msg, ""));
    }
}

function setDbFailMsg(ind) {
    var existing = $('#userdisplay').html();
    var msg = '<p class="error remove-bottom"><span class="strong">Error:</span> How embarrassing! We have been unable to connect to the database. Please contact an administrator.</p>';
    if (ind) {
        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
    } else {
        $('#userdisplay').html(existing.replace(msg, ""));
    }
}

function setDbVersionlMsg(ind) {
    var existing = $('#userdisplay').html();
    var msg = '<p class="error remove-bottom"><span class="strong">Error:</span> There is a database version mismatch between the GUI and the database. Please contact an administrator.</p>';
    if (ind) {
        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
    } else {
        $('#userdisplay').html(existing.replace(msg, ""));
    }
}

function setNoResultsMsg(ind) {
    var existing = $('#userdisplay').html();
    var msg = '<p class="info remove-bottom"><span class="strong">Info:</span> Sorry, your query didn\'t return any results. Please try again with new selection criteria.</p>';
    if (ind) {
        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
    } else {
        $('#userdisplay').html(existing.replace(msg, ""));
    }
}

function setPlotNotSupportedMsg(ind) {
    var existing = $('#userdisplay').html();
    var msg = '<p class="info remove-bottom"><span class="strong">Info:</span> Sorry, this plot is not yet supported. Please try again with a different plot.</p>';
    if (ind) {
        if (existing.indexOf(msg) === -1) $('#userdisplay').html(existing + msg);
    } else {
        $('#userdisplay').html(existing.replace(msg, ""));
    }
}

function setUserMsg(msg) {
    setCustomMsg(msg);
    if ($('#outerplot').html().indexOf("plots/no_graph.png") !== -1) {
        setPlotNotSupportedMsg(true);
    } else {
        setPlotNotSupportedMsg(false);
    }
    if ($('#outerplot').html().indexOf("plots/no_results.png") !== -1) {
        setNoResultsMsg(true);
    } else {
        setNoResultsMsg(false);
    }
    if ($('#outerplot').html().indexOf("plots/no_connection.png") !== -1) {
        setDbFailMsg(true);
    } else {
        setDbFailMsg(false);
    }
}

function server_list_has_content() {
    if ($("#servers option:selected").text() == '') {
        return false;
    } else {
        return true;
    }
}
///
/// Copyright 2014, 2015, 2016 Internet Corporation for Assigned Names and Numbers.
///
/// This Source Code Form is subject to the terms of the Mozilla Public
/// License, v. 2.0. If a copy of the MPL was not distributed with this
/// file, you can obtain one at https://mozilla.org/MPL/2.0/.
///
/// Developed by Sinodun IT (www.sinodun.com)
///

var sortByText = function (a, b) {
     return $.trim($(a).text()).localeCompare($.trim($(b).text()));
}

function parse_name( myid ) {
    // Need to process the server name as it could contain full stops.
    // Whilst these are allowed in identifiers if escaped their use complicates the code
    // So instead replace the full stop with a different character.
    // In CSS identifiers can contain only the characters [a-zA-Z0-9] and ISO 10646
    // characters U+00A0 and higher, plus the hyphen (-) and the underscore (_). 
    // We can't use with hyphen or underscore here, so use the Pound sign (U+00A3) which is valid.

    return myid.replace( /\./g, "Â£" );
}

function initNodeHtml(nodes_raw, enable_node_selection) {

    // Example data format
    // var nodes = [
    //   { server:     "Server-A",
    //     groups:     [{group_name: "Region-1", node_list: [{node_name: "Node-1", node_id: "1"}, {node_name: "Node-2", node_id: "2", node_country: "US", node_city: "City-1", node_sg: "Subgroup-1"}]},
    //                  {group_name: "Region-2", node_list: [{node_name: "Node-3", node_id: "3"}, {node_name: "Node-4", node_id: "4", node_country: "US", node_city: "City-1", node_sg: "Subgroup-2"}]}]},
    //   { server:     "L-root",
    //     groups:     [{group_name: "Europe",  node_list: [{node_name: "Node-5", node_id: "5"}, {node_name: "Node-6", node_id: "6", node_country: "US", node_city: "City-1", node_sg: "Subgroup-3"}}]},
    //                  {group_name: "America", node_list: [{node_name: "Node-7", node_id: "7"}, {node_name: "Node-8", node_id: "8", node_country: "US", node_city: "City-1", node_sg: "Subgroup-3"}}]}]}];

    nodes= jQuery.parseJSON(nodes_raw);

    // Append requires a full element and will close any open elements so need care when using it.
    $('#nodetabs').append("<div class='sixteen columns' id='groupcontent'></div>");
    $('#nodetabs').append("<br class='clear'>");
    $('#nodetabs').append("<div class='sixteen columns' id='nodecontent'>");
    $('#srvr_select').append("<a style='font-weight:bold; font-size:80%; text-decoration: none; padding-left: 20px; padding-right: 5px;'>Group by:</a>");
    $('#srvr_select').append("<input type='radio'          id='ng_country'  name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_country'  >Country </label>");
    $('#srvr_select').append("<input type='radio'          id='ng_city'     name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_city'     >City </label>");
    $('#srvr_select').append("<input type='radio'          id='ng_instance' name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_instance' >Instance </label>");
    if (enable_node_selection == 1) {$('#srvr_select').append("<input type='radio'          id='ng_none'     name='node_grouping' onclick='ngChanged()' ><label style='font-weight:bold;' for='ng_none'     >Node </label>");}

    // Check for no servers is made earlier, so just check for nodes here
    // if (nodes_raw = "[]") {
    //     alert("No nodes found in the database");
    // }

    //*** SERVERS ***//
    // For each server, construct the groups.
    for (var i = 0; i < nodes.length; i++) {
        var server_tmp = nodes[i].server;
        var server     = parse_name(server_tmp);
        $('#groupcontent'       ).append(  "<div class='hidden' id='" + server + "'>");
        $('#' + server          ).append(  "<div class='Tabs'   id='" + server + "_tabs'>");
        $('#' + server + '_tabs').append(  "<ul                 id='" + server +"_tabs_ul'>");
        //***  GROUPS (REGIONS) ***//
        // For each group, add a tab and fill in the nodes
        for (var j = 0; j < nodes[i].groups.length; j++) {
            var group_tmp = nodes[i].groups[j].group_name;
            var group     = parse_name(group_tmp);
            var group_server       = group + "_" + server;
            var group_server_basic = group_server;
            // Create the top tab with a list entry
            $('#' + server + '_tabs_ul').append("<li id='li_" + group_server + "' onclick='gpTab(\"" + group_server + "\")'>  <a>" + group_tmp + "<img id='cb_" + group_server +"_img' src='images/all.png' alt='all selected' height='10' width='10'></a></li>");
            //Now the node content divs
            for (var node_grouping_options = 0; node_grouping_options <=3; node_grouping_options++) {
                if (node_grouping_options == 1)      {group_server = group + "_" + server + "_subgroup";}
                else if (node_grouping_options == 2) {group_server = group + "_" + server + "_city";}
                else if (node_grouping_options == 3) {group_server = group + "_" + server + "_country";}
                $('#nodecontent'                 ).append("<div class='group_showing'   id='" + group_server + "'>");
                $('#' + group_server             ).append("<div class='allnone'         id='" + group_server + "_allnone'>");
                $('#' + group_server             ).append("<div class='node_cbs'        id='" + group_server + "_node_cbs_id'>");
                // Selection buttons
                $('#' + group_server + '_allnone').append("<a style='font-weight:bold;'>Actions:</a>");
                if (j == -1) {
                    $('#' + group_server + '_allnone').append("<input type='button' id='selectAllBtn2'  value='Select all nodes'    title='Select all available nodes for plotting' onclick='selectAll(\"cb_" + group_server_basic + "\")'>"); 
                    $('#' + group_server + '_allnone').append("<input type='button' id='selectNoneBtn2' value='De-select all nodes' title='De-select all available nodes'           onclick='selectNone(\"cb_" + group_server_basic + "\")'><hr>");
                } else {
                    $('#' + group_server + '_allnone').append("<input type='button' id='selectOnlyRegionBtn' value='Select only " + group + "' title='Select only the nodes in this region for plotting'   onclick='selectOnly(\"cb_" + group_server_basic + "\")'> ");
                    $('#' + group_server + '_allnone').append("<input type='button' id='selectAllRegionBtn'  value='Include all " + group + "' title='Include all the nodes in this region for plotting'   onclick='selectAll(\"cb_"  + group_server_basic + "\")'> ");
                    $('#' + group_server + '_allnone').append("<input type='button' id='selectNoneRegionBtn' value='Exclude all " + group + "' title='De-select all the nodes in this region for plotting' onclick='selectNone(\"cb_" + group_server_basic + "\")'><hr> ");
                }
                //*** NODES AND GROUPING OPTIONS***//
                for (var k = 0; k < nodes[i].groups[j].node_list.length; k++) {
                    var node                 = nodes[i].groups[j].node_list[k]
                    var node_id_group_server = node.node_id + "_" + group_server_basic;
                    if (node_grouping_options == 0) {
                        $('#' + group_server + '_node_cbs_id').append("<input type='checkbox' class='nodeselection' id='" + node_id_group_server + "' name='cb_" + group_server + "' onclick='selectNode(\"" + node_id_group_server + "\")'>");
                        $('#' + group_server + '_node_cbs_id').append("<label for='" + node_id_group_server + "' title='Toggle node selection for " + node.node_name + "\n Country: " + node.node_country + "\n City: " + node.node_city + "\n Instance: " + node.node_sg + "'>" + node.node_name + "</label>");
                    } else {
                        if (node_grouping_options == 1) {
                            var node_groupby_label   = node.node_sg;
                            var node_groupby_class   = 'subgroupselection';
                        }
                        if (node_grouping_options == 2) {
                            var node_groupby_label   = node.node_city;
                            var node_groupby_class   = 'cityselection';
                        }   
                        if (node_grouping_options == 3) {
                            var node_groupby_label   = node.node_country;
                            var node_groupby_class   = 'countryselection';
                        }
                        // use tilde not underscores as delimiter so other code doesn't trigger off this
                        var node_groupby_group_server = node_groupby_label + "~" + group + "~" + server;
                        if($("input[type='checkbox'][id='" + node_groupby_group_server + "']").length == 0) {
                            $('#' + group_server + '_node_cbs_id').append("<label for='" + node_groupby_group_server + "' title='Toggle node selection for:\n" + node.node_name + "'>" + node_groupby_label + " <img id='cb_" + node_groupby_group_server +"_img' src='images/all.png' alt='all selected' height='10' width='10'>");
                            $('#' + group_server + '_node_cbs_id').append("<input type='checkbox' class='" + node_groupby_class + "' id='" + node_groupby_group_server + "' name='cb-" + node_groupby_group_server + "' value='" + node_id_group_server + "' onclick='selectGrouping(this)'></label>");
                        } else {
                            var my_label = $("label[for='" + node_groupby_group_server + "']").attr("title");
                            $("label[for='" + node_groupby_group_server + "']").attr("title", my_label + "\n" + node.node_name);
                            var my_cb = $("input[type='checkbox'][id='" + node_groupby_group_server + "']");
                            $.each(my_cb, function() {
                               this.value = this.value + "~" + node_id_group_server; 
                            });
                        }
                    }
                }
                if (node_grouping_options != 0) {
                    // We must sort the country, city and instances alphabetically
                    var sorted = $('#' + group_server + '_node_cbs_id label').sort(sortByText);
                    $('#' + group_server + '_node_cbs_id').append(sorted);
                }
            }
        }
    }
}

function ngChanged() {
    var active_group = $( "li.activeGpTab" );
    $.each(active_group, function() {
        var group_name = this.id.substring(3, this.id.length);
        gpTab(group_name);
    });
}

function setServersGroups() {
    // sets window.servers to the set of server names,
    // and window.groups to the set of group names.
    // nodeselection checkbox name format: 'node-id_group_server'
    window.servers = {};
    window.groups = {};
    $("input[type='checkbox'].nodeselection").each(function(){
        var s = $(this).attr('name').split('_')[2];
        var g = $(this).attr('name').split('_')[1];
        window.servers[s] = true;
        window.groups[g + "_" + s] = true;
    });
}

function selectSisterGrouping(grp_class) {
    $("input[type='checkbox']." + grp_class).each(function(){
         node_list = this.value.split("~");
         var all = true;
         var none = true;
         $.each(node_list, function(i, value){
             var node_cb = $("#" + value);
             if($(node_cb).is(':checked')){
                 none = false;
             }else{
                 all = false;
             }
         });
         if (all == true) {
             $(this).prop('checked', true);
             $("label[for='" + this.id + "']").css("color", '#000');
             document.getElementById("cb_" + this.id + "_img").src="images/all.png";
             document.getElementById("cb_" + this.id + "_img").alt="all selected";
         } else if (none == true) {
             $(this).prop('checked', false);
             $("label[for='" + this.id + "']").css("color", '#999');
             document.getElementById("cb_" + this.id + "_img").src="images/none.png";
             document.getElementById("cb_" + this.id + "_img").alt="none selected";
         } else {
             $(this).prop('checked', true);
             //$("label[for='" + this.id + "']").css("color", '#0072B2');
             $("label[for='" + this.id + "']").css("color", '#000');
             var temp = $("#cb_" + this.id + "_img");
             document.getElementById("cb_" + this.id + "_img").src="images/some.png";
             document.getElementById("cb_" + this.id + "_img").alt="some selected";
         }
     });
}


function selectSister(idList, ckd) {
    // finds all checkboxes where their node id
    // is in idList and sets their checked status
    // to ckd (true / false)
    $.each(idList, function(i, value){
        var ids = value.split('_');
        var nd = ids[0];
        var gp = ids[1];
        var srvr = ids[2];
        for(gs in window.groups){
            g = gs.split('_')[0];
            s = gs.split('_')[1];
            if((s === srvr) /* && (g !== gp)*/){
                var sisterId = nd + '_' + g + '_' + srvr;
                var sister = $("#" + sisterId);
                if(sister){
                    sister.prop('checked', ckd);
                }
            }
        }
    });
    
    // update to also set the cbs in the grouping tabs
    var grouping = ["subgroupselection", "cityselection", "countryselection"]; 
    for (i = 0; i < 3; i++) { 
        selectSisterGrouping(grouping[i]);
    }

    // set group tab images to reflect
    // whether all, some or no checkboxes
    // are selected
    initnodetabs();
}

function select(cbGp, ckd) {
    // for all checkboxes with the name cbGp
    // set checked status to ckd (true/false)
    var idList = new Array();
    $("input[type='checkbox'][name=" + cbGp + "]").each(function(){
        $(this).prop('checked', ckd);
        idList.push($(this).attr('id'));
    });
    // ensure any sister nodes are
    // updated to maintain consistency
    selectSister(idList, ckd);
}

function updateAll(ckd) {
    // for all checkboxes 
    // set checked status to ckd (true/false)
    var idList = new Array();
    $("input[type='checkbox']").each(function(){
        $(this).prop('checked', ckd);
        idList.push($(this).attr('id'));
    });
    // ensure any sister nodes are
    // updated to maintain consistency
    selectSister(idList, ckd);
}

function selectAll(cbGp) {
    // set all checkboxes with the
    // name cbGp to checked: true
    select(cbGp,true);
}

function selectNone(cbGp) {
    // set all checkboxes with the
    // name cdGp to checked: false
    select(cbGp,false);
}

function selectOnly(cbGp) {
    // set all checkboxes to false then 
	// set all with
    // name cdGp to checked: true
	updateAll(false)
    select(cbGp,true);
}


function selectNode(ckbxid) {
    // ensure any sister nodes are
    // updated to maintain consistency
    idList = new Array();
    idList.push(ckbxid);
    selectSister(idList, $("#" + ckbxid).is(':checked'));
}

function selectGrouping(item) {
    idList = item.value.split("~");
    selectSister(idList, $(item).is(':checked'));
    var my_subgroup = item.id.split("~")[0];
    var my_server   = item.id.split("~")[2];
    $("input[type='checkbox']." + item.class).each(function(){
        var this_subgroup = this.id.split("~")[0];
        var this_server = this.id.split("~")[2];
        if (my_server == this_server && my_subgroup == this_subgroup) {
            this.checked = $(item).is(':checked');
        }
    });
}

function selectSome(cbGs) {
    // determine whether all, some or no
    // checkboxes are checked where their
    // name is cbGs and then update the
    // group tab image accordingly
    var none = true;
    var all = true;
    
    $("input[type='checkbox'][name=" + cbGs + "]").each(function() {
        if($(this).is(':checked')){
            none = false;
        }else{
            all = false;
        }
    });
    
    if(none){
        $("#" + cbGs + "_img").attr("src", "images/none.png");
        $("#" + cbGs + "_img").attr("alt", "none selected"); 
    }else if(all){
        $("#" + cbGs + "_img").attr("src", "images/all.png");
        $("#" + cbGs + "_img").attr("alt", "all selected");
    }else{
        $("#" + cbGs + "_img").attr("src", "images/some.png");
        $("#" + cbGs + "_img").attr("alt", "some selected");
    }
    
}

function initnodetabs(){
    // for each group tab, set the image
    // which indicates whether all, some
    // or no checkboxes are selected

    for(var gs in window.groups) {
        selectSome('cb_' + gs);
    }
    enableGenerate(true);
}

function gpTab(gpnm) {
    // set active group tab, display associated
    // node content and hide old node content from view
    $("li.activeGpTab").removeClass("activeGpTab");
    $("#li_" + gpnm).addClass("activeGpTab");
    old_content = $(".group_showing");
    old_content.removeClass("group_showing");
    old_content.addClass("hidden");
    if ($('#ng_instance').prop('checked') === true) gpnm = gpnm + "_subgroup";
    if ($('#ng_city').prop('checked')     === true) gpnm = gpnm + "_city";
    if ($('#ng_country').prop('checked')  === true) gpnm = gpnm + "_country";
    $("#" + gpnm).removeClass("hidden");
    $("#" + gpnm).addClass("group_showing");
}

function serverTab() {
    // display associated group tabs for selected
    // server and hide old group tabs from view
    gptabsid_tmp = $("#servers option:selected").text();
    gptabsid = parse_name(gptabsid_tmp);
    old_content = $("div.server_showing");
    old_content.removeClass('server_showing');
    old_content.addClass('hidden');
    $("#" + gptabsid).removeClass('hidden');
    $("#" + gptabsid).addClass('server_showing');
    gpTab('All_' +  gptabsid);
    for(var gs in window.groups) {
        s = gs.split('_')[1];
        if(s === gptabsid){
            selectAll('cb_' + gs);
        }else{
            selectNone('cb_' + gs);
        }
        selectSome('cb_' + gs);
    }
}
/*
* Skeleton V1.2
* Copyright 2011, Dave Gamache
* www.getskeleton.com
* Free to use under the MIT license.
* http://www.opensource.org/licenses/mit-license.php
* 6/20/2012
*/


/* Table of Content
==================================================
	#Reset & Basics
	#Basic Styles
	#Site Styles
	#Typography
	#Links
	#Lists
	#Images
	#Buttons
	#Forms
	#Misc */


/* #Reset & Basics (Inspired by E. Meyers)
================================================== */
	html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video {
		margin: 0;
		padding: 0;
		border: 0;
		font-size: 100%;
		font: inherit;
		vertical-align: baseline; }
	article, aside, details, figcaption, figure, footer, header, hgroup, menu, nav, section {
		display: block; }
	body {
		line-height: 1; }
	ol, ul {
		list-style: none; }
	blockquote, q {
		quotes: none; }
	blockquote:before, blockquote:after,
	q:before, q:after {
		content: '';
		content: none; }
	table {
		border-collapse: collapse;
		border-spacing: 0; }


/* #Basic Styles
================================================== */
	body {
		background: #fff;
		font: 14px/21px "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif;
		color: #444;
		-webkit-font-smoothing: antialiased; /* Fix for webkit rendering */
		-webkit-text-size-adjust: 100%;
 }


/* #Typography
================================================== */
	h1, h2, h3, h4, h5, h6 {
		color: #181818;
		font-family: "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif;
		font-weight: normal; }
	h1 a, h2 a, h3 a, h4 a, h5 a, h6 a { font-weight: inherit; }
	h1 { font-size: 46px; line-height: 50px; margin-bottom: 14px;}
	h2 { font-size: 35px; line-height: 40px; margin-bottom: 10px; }
	h3 { font-size: 28px; line-height: 34px; margin-bottom: 8px; }
	h4 { font-size: 21px; line-height: 30px; margin-bottom: 4px; }
	h5 { font-size: 17px; line-height: 24px; }
	h6 { font-size: 14px; line-height: 21px; }
	.subheader { color: #777; }

	p { margin: 0 0 20px 0; }
	p img { margin: 0; }
	p.lead { font-size: 21px; line-height: 27px; color: #777;  }

	em { font-style: italic; }
	strong { font-weight: bold; color: #333; }
	small { font-size: 80%; }

/*	Blockquotes  */
	blockquote, blockquote p { font-size: 17px; line-height: 24px; color: #777; font-style: italic; }
	blockquote { margin: 0 0 20px; padding: 9px 20px 0 19px; border-left: 1px solid #ddd; }
	blockquote cite { display: block; font-size: 12px; color: #555; }
	blockquote cite:before { content: "\2014 \0020"; }
	blockquote cite a, blockquote cite a:visited, blockquote cite a:visited { color: #555; }

	hr { border: solid #ddd; border-width: 1px 0 0; clear: both; margin: 10px 0 30px; height: 0; }


/* #Links
================================================== */
	a, a:visited { color: #333; text-decoration: underline; outline: 0; }
	a:hover, a:focus { color: #000; }
	p a, p a:visited { line-height: inherit; }


/* #Lists
================================================== */
	ul, ol { margin-bottom: 20px; }
	ul { list-style: none outside; }
	ol { list-style: decimal; }
	ol, ul.square, ul.circle, ul.disc { margin-left: 30px; }
	ul.square { list-style: square outside; }
	ul.circle { list-style: circle outside; }
	ul.disc { list-style: disc outside; }
	ul ul, ul ol,
	ol ol, ol ul { margin: 4px 0 5px 30px; font-size: 90%;  }
	ul ul li, ul ol li,
	ol ol li, ol ul li { margin-bottom: 6px; }
	li { line-height: 18px; margin-bottom: 12px; }
	ul.large li { line-height: 21px; }
	li p { line-height: 21px; }

/* #Images
================================================== */

	img.scale-with-grid {
		max-width: 100%;
		height: auto; }


/* #Buttons
================================================== */

	.button,
	button,
	input[type="submit"],
	input[type="reset"],
	input[type="text"],
	input[type="button"] {
		background: #eee; /* Old browsers */
		background: #eee -moz-linear-gradient(top, rgba(255,255,255,.2) 0%, rgba(0,0,0,.2) 100%); /* FF3.6+ */
		background: #eee -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.2)), color-stop(100%,rgba(0,0,0,.2))); /* Chrome,Safari4+ */
		background: #eee -webkit-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Chrome10+,Safari5.1+ */
		background: #eee -o-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Opera11.10+ */
		background: #eee -ms-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* IE10+ */
		background: #eee linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* W3C */
	  border: 1px solid #aaa;
	  border-top: 1px solid #ccc;
	  border-left: 1px solid #ccc;
	  -moz-border-radius: 3px;
	  -webkit-border-radius: 3px;
	  border-radius: 3px;
	  color: #444;
	  display: inline-block;
	  font-size: 11px;
	  font-weight: bold;
	  text-decoration: none;
	  text-shadow: 0 1px rgba(255, 255, 255, .75);
	  cursor: pointer;
	  margin-bottom: 20px;
	  line-height: normal;
	  padding: 8px 10px;
	  font-family: "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif; }

	.button:hover,
	button:hover,
	input[type="submit"]:hover,
	input[type="reset"]:hover,
	input[type="button"]:hover {
		color: #222;
		background: #ddd; /* Old browsers */
		background: #ddd -moz-linear-gradient(top, rgba(255,255,255,.3) 0%, rgba(0,0,0,.3) 100%); /* FF3.6+ */
		background: #ddd -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.3)), color-stop(100%,rgba(0,0,0,.3))); /* Chrome,Safari4+ */
		background: #ddd -webkit-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Chrome10+,Safari5.1+ */
		background: #ddd -o-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Opera11.10+ */
		background: #ddd -ms-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* IE10+ */
		background: #ddd linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* W3C */
	  border: 1px solid #888;
	  border-top: 1px solid #aaa;
	  border-left: 1px solid #aaa; }

	.button:active,
	button:active,
	input[type="submit"]:active,
	input[type="reset"]:active,
	input[type="button"]:active {
		border: 1px solid #666;
		background: #ccc; /* Old browsers */
		background: #ccc -moz-linear-gradient(top, rgba(255,255,255,.35) 0%, rgba(10,10,10,.4) 100%); /* FF3.6+ */
		background: #ccc -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.35)), color-stop(100%,rgba(10,10,10,.4))); /* Chrome,Safari4+ */
		background: #ccc -webkit-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Chrome10+,Safari5.1+ */
		background: #ccc -o-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Opera11.10+ */
		background: #ccc -ms-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* IE10+ */
		background: #ccc linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* W3C */ }

	.button.full-width,
	button.full-width,
	input[type="submit"].full-width,
	input[type="reset"].full-width,
	input[type="button"].full-width {
		width: 100%;
		padding-left: 0 !important;
		padding-right: 0 !important;
		text-align: center; }

	/* Fix for odd Mozilla border & padding issues */
	button::-moz-focus-inner,
	input::-moz-focus-inner {
    border: 0;
    padding: 0;
	}


/* #Forms
================================================== */

	form {
		margin-bottom: 20px; }
	fieldset {
		margin-bottom: 20px; }
	input[type="text"],
	input[type="password"],
	input[type="email"],
        input[type="date"],
        input[type="datetime-local"],
        input[type="time"],
	textarea,
	select {
		border: 1px solid #ccc;
		padding: 6px 4px;
		outline: none;
		-moz-border-radius: 2px;
		-webkit-border-radius: 2px;
		border-radius: 2px;
		font: 13px "HelveticaNeue", "Helvetica Neue", Helvetica, Arial, sans-serif;
		color: #777;
		margin: 0;
		width: 210px;
		max-width: 100%;
		display: inline;
		margin-bottom: 20px;
		background: #fff; }
	select {
		padding: 0; }
	
	input[type="text"]:focus,
	input[type="password"]:focus,
	input[type="email"]:focus,
        input[type="date"]:focus,
        input[type="datetime-local"]:focus,
        input[type="time"]:focus,
	textarea:focus {
		border: 1px solid #aaa;
 		color: #444;
 		-moz-box-shadow: 0 0 3px rgba(0,0,0,.2);
		-webkit-box-shadow: 0 0 3px rgba(0,0,0,.2);
		box-shadow:  0 0 3px rgba(0,0,0,.2); }
	textarea {
		min-height: 60px; }
	label,
	legend {
		display: inline;
		font-weight: bold;
		font-size: 13px;  }
	select{
		width: 220px; }
	input[type="checkbox"] {
		display: inline; }
	label span,
	legend span {
		font-weight: normal;
		font-size: 13px;
		color: #444; }

/* #Misc
================================================== */
	.remove-bottom { margin-bottom: 0 !important; }
	.half-bottom { margin-bottom: 10px !important; }
	.add-bottom { margin-bottom: 20px !important; }


/*
* Skeleton V1.2
* Copyright 2011, Dave Gamache
* www.getskeleton.com
* Free to use under the MIT license.
* http://www.opensource.org/licenses/mit-license.php
* 6/20/2012
*/

/* Table of Content
==================================================
	#Site Styles
	#Page Styles
	#Media Queries
	#Font-Face */

/* #Site Styles
================================================== */

.strong {
    font-weight: bold;
}

.logo {
    #padding: 2px 2px;
    color: #fff;
    background-color: #fff;
    border: 3px solid #5894b0;
    -webkit-border-radius: 4px;
    -moz-border-radius: 4px;
    border-radius: 4px;    
}

.info {
    padding: 2px 8px;
    color: #3a87ad;
    background-color: #d9edf7;
    border: 1px solid #bce8f1;
    -webkit-border-radius: 4px;
    -moz-border-radius: 4px;
    border-radius: 4px;    
}

.warning {
    padding: 2px 8px;
    color: #c09853;
    background-color: #fcf8e3;
    border: 1px solid #fbeed5;
    -webkit-border-radius: 4px;
    -moz-border-radius: 4px;
    border-radius: 4px;
}

.error {
    padding: 2px 8px;
    color: #b94a48;
    background-color: #f2dede;
    border: 1px solid #eed3d7;
    -webkit-border-radius: 4px;
    -moz-border-radius: 4px;
    border-radius: 4px;
}

.hidden {
    display: none !important;
}

/*input, textarea {
  /* stop iOS mobile safari from overriding my border radius settings */
  /*-webkit-appearance: none;*/
  -webkit-border-radius: 0;
}*/

ul.discs {
    list-style-type: disc;
}

ul.circles {
    list-style-type: circle;
}

/* #Page Styles
================================================== */

#helpimg {
    position: relative;
    top: 2px;
    left: 8px;
}

#generate:not([disabled]){
    background: #5894b0; /* Old browsers */
    background: #5894b0 -moz-linear-gradient(top, rgba(255,255,255,.2) 0%, rgba(0,0,0,.2) 100%); /* FF3.6+ */
    background: #5894b0 -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.2)), color-stop(100%,rgba(0,0,0,.2))); /* Chrome,Safari4+ */
    background: #5894b0 -webkit-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Chrome10+,Safari5.1+ */
    background: #5894b0 -o-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Opera11.10+ */
    background: #5894b0 -ms-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* IE10+ */
    background: #5894b0 linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* W3C */
    color: white;
    text-shadow: none;
}

#generate:not([disabled]):hover {
    background: #4e8aa6; /* Old browsers */
    background: #4e8aa6 -moz-linear-gradient(top, rgba(255,255,255,.3) 0%, rgba(0,0,0,.3) 100%); /* FF3.6+ */
    background: #4e8aa6 -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.3)), color-stop(100%,rgba(0,0,0,.3))); /* Chrome,Safari4+ */
    background: #4e8aa6 -webkit-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Chrome10+,Safari5.1+ */
    background: #4e8aa6 -o-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* Opera11.10+ */
    background: #4e8aa6 -ms-linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* IE10+ */
    background: #4e8aa6 linear-gradient(top, rgba(255,255,255,.3) 0%,rgba(0,0,0,.3) 100%); /* W3C */
}

#generate:not([disabled]):active {
    background: #42748b -moz-linear-gradient(top, rgba(255,255,255,.35) 0%, rgba(10,10,10,.4) 100%); /* FF3.6+ */
    background: #42748b -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.35)), color-stop(100%,rgba(10,10,10,.4))); /* Chrome,Safari4+ */
    background: #42748b -webkit-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Chrome10+,Safari5.1+ */
    background: #42748b -o-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* Opera11.10+ */
    background: #42748b -ms-linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* IE10+ */
    background: #42748b linear-gradient(top, rgba(255,255,255,.35) 0%,rgba(10,10,10,.4) 100%); /* W3C */
}

input[type="button"][disabled] {
    color: GrayText;
    cursor: inherit;
}

input[type="button"][disabled]:hover {
    color: GrayText;
    cursor: inherit;
    background: #eee; /* Old browsers */
    background: #eee -moz-linear-gradient(top, rgba(255,255,255,.2) 0%, rgba(0,0,0,.2) 100%); /* FF3.6+ */
    background: #eee -webkit-gradient(linear, left top, left bottom, color-stop(0%,rgba(255,255,255,.2)), color-stop(100%,rgba(0,0,0,.2))); /* Chrome,Safari4+ */
    background: #eee -webkit-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Chrome10+,Safari5.1+ */
    background: #eee -o-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* Opera11.10+ */
    background: #eee -ms-linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* IE10+ */
    background: #eee linear-gradient(top, rgba(255,255,255,.2) 0%,rgba(0,0,0,.2) 100%); /* W3C */
    border: 1px solid #aaa;
    border-top: 1px solid #ccc;
    border-left: 1px solid #ccc;
}

#timeTabsAdv {
    padding-left: 14px;
    margin-bottom: 8px;
}

#timeTabsAdv label,
#srvr_select label,
#plotselection label,
#start,
#stop,
#servers,
#plotType {
    display: inline-block;
    margin-bottom: 0;
    font-size: 80%;
}

#start,
#stop,
#plotType {
    width: 176px;
    padding: 0;
}

#setToday {
    margin: 0 0 6px 32px;
    padding: 2px 8px;
}

#plotType {
    margin: 5px 0 0 0;
}

#generate {
    margin: 4px 0 0 0;
    padding: 2px 4px;
    font-size: 80%;
    width: 200px;
}


#timeTabsAdv label {
    width: 26px;
}

#timeTabsAdv label,
#srvr_select label,
#plotselection label {
    text-align: right;
}

.Tabs ul {
    padding: 0px;
    margin: 0px;
    margin-left: 10px;
    list-style-type: none;
}

.Tabs ul li {
    display: inline-block;
    float: left;
    height: 24px;
    cursor: pointer;
    margin-bottom: 21px;
}

.Tabs ul li a {
    position: relative;
    margin-top: 16px;
    display: block;
    margin-left: 6px;
    line-height: 28px;
    padding-left: 10px;
    padding-right: 10px;
    background: #fff;
    z-index: 9999;
    border: 1px solid #ccc;
    -moz-border-radius-topleft: 3px;
    border-top-left-radius: 3px;
    -moz-border-radius-topright: 3px;
    border-top-right-radius: 3px;
    color: #444;
    text-decoration: none;
    font-weight: bold;
}

.Tabs ul li.activeGpTab a {
    background: #eee;
    color: #000;
    position: relative;
    top: 1px;
    border-bottom: 0px;
}

.Tabs ul li a:hover {
    color: #222;
    background: #ccc;
}

.Tabs #groupcontent,
.Tabs .group_showing {
    padding: 0 15px;
    clear:both;
    line-height:19px;
    position: relative;
    top: 20px;
    z-index: 5;
    height: 150px;
    overflow: hidden;
}

.Tabs ul li a img {
    position: relative;
    top: 1px;
    left: 4px;
}

.group_showing {
    display: block;
    height: 210px;
    /*overflow-y: scroll;*/
    padding: 5px 0 0 20px;
    border: 1px solid #ccc;
    -moz-border-radius: 3px;
    -webkit-border-radius: 3px;
    border-radius: 3px;   
}

.node_cbs {
    overflow-y: scroll;
    display: block;
    height: 169px;
}

#outerplot {
    min-height: 528px;
    height: auto !important;
    height: 528px;
}

#plot {
    text-align: center;
}

#plot hr {
    margin-bottom: 10px;
}

#lextend {
    -moz-border-radius: 3px 0 0 3px;
    -webkit-border-radius: 3px 0 0 3px;
    border-radius: 3px 0 0 3px;
}

#rextend {
    -moz-border-radius: 0 3px 3px 0;
    -webkit-border-radius: 0 3px 3px 0;
    border-radius: 0 3px 3px 0;
}

.group_showing {
    margin-bottom: 20px;
}


.group_showing label {
    color: #000;
    display: inline;
    font-family: "New Courier", Monospace;
    cursor: pointer;
    -moz-border-radius: 5px;
    -webkit-border-radius: 5px;
    border-radius: 5px;
}

.node_cbs label {
    color: #ccc;
    display: inline-block;
    margin: 0 5px 5px 0;
    padding: 4px;
    font-family: "New Courier", Monospace;
    cursor: pointer;
    background: #eee;
    -moz-border-radius: 5px;
    -webkit-border-radius: 5px;
    border-radius: 5px;
}

.group_showing label:hover {
    background: #ddd;
}

.mycolumn {
    float: left;
    display: inline;
}

#lextend,
#rextend,
#larrow,
#rarrow {
    width: 27px;
    height: 48px;
    margin-bottom: 4px !important;
}

#timeTabsBasic label.button,
#lextend,
#rextend,
#larrow,
#rarrow {
    /* remove color gradient from these buttons */
    background: #eee;
}

#timeTabsBasic label.button,
#larrow,
#rarrow {
    -moz-border-radius: 0;
    -webkit-border-radius: 0;
    border-radius: 0;
}

#timeTabsBasic label.button,
#timeTabsBasic input[type="button"] {
    margin: 0;
    padding: 4px;
    font-size: 80%;
}

#timeTabsBasic label.button {
    text-align: center;
    width: 47px;
}

#timeTabsBasic input[type="radio"],
.node_cbs input[type="checkbox"] {
   display: none;
}

#timeTabsBasic input[type="radio"]:checked + label.button {
    border: 1px solid #666;
    background: #ccc; /* Old browsers */
}


.node_cbs input[type="checkbox"]:checked + label {
    color: #999;
}

.group_showing input[type="checkbox"]:checked + label {
    color: #000;
}

.group_showing input[type="button"] {
    margin: 0 0 2px 16px;
    padding: 2px 8px;
}

.group_showing hr {  
margin: 5px 0px 10px; 
}

.allnone {
    margin: 2px 2px 5px 0;
}

.basicadv,
.allnone {
    font-size: 80% !important;
    text-align: left;
}

.basicadv a,
.allnone a {
    text-decoration: none;
    cursor: pointer;
}

#timedisplay {
    text-align: right;
    margin-top: 26px;
}

#userdisplay {
    margin-top: 1px;
    min-height: 13px;
    height: auto !important;
}

#timedisplay p,
#userdisplay p {
    font-size: 80%;
}

#opts {
    text-align: right;
}

#opts input[type="checkbox"] {
    -webkit-appearance: checkbox;
}

#opts input[type="radio"],
#radioplt input[type="radio"] {
    -webkit-appearance: radio;
}

#opts input[type="checkbox"] + label,
#opts input[type="radio"] + label,
#radioplt input[type="radio"] + label {
    display: inline;
}

#copyright {
    text-align: center;
}

/* #Media Queries
================================================== */

	/* Smaller than standard 960 (devices and browsers) */
	@media only screen and (max-width: 959px) {}

	/* Tablet Portrait size to standard 960 (devices and browsers) */
	@media only screen and (min-width: 768px) and (max-width: 959px) {}

	/* All Mobile Sizes (devices and browser) */
	@media only screen and (max-width: 767px) {}

	/* Mobile Landscape Size to Tablet Portrait (devices and browsers) */
	@media only screen and (min-width: 480px) and (max-width: 767px) {}

	/* Mobile Portrait Size to Mobile Landscape Size (devices and browsers) */
	@media only screen and (max-width: 479px) {}


/* #Font-Face
================================================== */
/* 	This is the proper syntax for an @font-face file
		Just create a "fonts" folder at the root,
		copy your FontName into code below and remove
		comment brackets */

/*	@font-face {
	    font-family: 'FontName';
	    src: url('../fonts/FontName.eot');
	    src: url('../fonts/FontName.eot?iefix') format('eot'),
	         url('../fonts/FontName.woff') format('woff'),
	         url('../fonts/FontName.ttf') format('truetype'),
	         url('../fonts/FontName.svg#webfontZam02nTh') format('svg');
	    font-weight: normal;
	    font-style: normal; }
*/
/*
* Skeleton V1.2
* Copyright 2011, Dave Gamache
* www.getskeleton.com
* Free to use under the MIT license.
* http://www.opensource.org/licenses/mit-license.php
* 6/20/2012
*/


/* Table of Contents
==================================================
    #Base 960 Grid
    #Tablet (Portrait)
    #Mobile (Portrait)
    #Mobile (Landscape)
    #Clearing */



/* #Base 960 Grid
================================================== */

    .container                                  { position: relative; width: 960px; margin: 0 auto; padding: 0; }
    .container .column,
    .container .columns                         { float: left; display: inline; margin-left: 10px; margin-right: 10px; }
    .row                                        { margin-bottom: 20px; }

    /* Nested Column Classes */
    .column.alpha, .columns.alpha               { margin-left: 0; }
    .column.omega, .columns.omega               { margin-right: 0; }

    /* Base Grid */
    .container .one.column,
    .container .one.columns                     { width: 40px;  }
    .container .two.columns                     { width: 100px; }
    .container .three.columns                   { width: 160px; }
    .container .four.columns                    { width: 220px; }
    .container .five.columns                    { width: 280px; }
    .container .six.columns                     { width: 340px; }
    .container .seven.columns                   { width: 400px; }
    .container .eight.columns                   { width: 460px; }
    .container .nine.columns                    { width: 520px; }
    .container .ten.columns                     { width: 580px; }
    .container .eleven.columns                  { width: 640px; }
    .container .twelve.columns                  { width: 700px; }
    .container .thirteen.columns                { width: 760px; }
    .container .fourteen.columns                { width: 820px; }
    .container .fifteen.columns                 { width: 880px; }
    .container .sixteen.columns                 { width: 940px; }

    .container .one-third.column                { width: 300px; }
    .container .two-thirds.column               { width: 620px; }

    /* Offsets */
    .container .offset-by-one                   { padding-left: 60px;  }
    .container .offset-by-two                   { padding-left: 120px; }
    .container .offset-by-three                 { padding-left: 180px; }
    .container .offset-by-four                  { padding-left: 240px; }
    .container .offset-by-five                  { padding-left: 300px; }
    .container .offset-by-six                   { padding-left: 360px; }
    .container .offset-by-seven                 { padding-left: 420px; }
    .container .offset-by-eight                 { padding-left: 480px; }
    .container .offset-by-nine                  { padding-left: 540px; }
    .container .offset-by-ten                   { padding-left: 600px; }
    .container .offset-by-eleven                { padding-left: 660px; }
    .container .offset-by-twelve                { padding-left: 720px; }
    .container .offset-by-thirteen              { padding-left: 780px; }
    .container .offset-by-fourteen              { padding-left: 840px; }
    .container .offset-by-fifteen               { padding-left: 900px; }



/* #Tablet (Portrait)
================================================== */

    /* Note: Design for a width of 768px */

    @media only screen and (min-width: 768px) and (max-width: 959px) {
        .container                                  { width: 768px; }
        .container .column,
        .container .columns                         { margin-left: 10px; margin-right: 10px;  }
        .column.alpha, .columns.alpha               { margin-left: 0; margin-right: 10px; }
        .column.omega, .columns.omega               { margin-right: 0; margin-left: 10px; }
        .alpha.omega                                { margin-left: 0; margin-right: 0; }

        .container .one.column,
        .container .one.columns                     { width: 28px; }
        .container .two.columns                     { width: 76px; }
        .container .three.columns                   { width: 124px; }
        .container .four.columns                    { width: 172px; }
        .container .five.columns                    { width: 220px; }
        .container .six.columns                     { width: 268px; }
        .container .seven.columns                   { width: 316px; }
        .container .eight.columns                   { width: 364px; }
        .container .nine.columns                    { width: 412px; }
        .container .ten.columns                     { width: 460px; }
        .container .eleven.columns                  { width: 508px; }
        .container .twelve.columns                  { width: 556px; }
        .container .thirteen.columns                { width: 604px; }
        .container .fourteen.columns                { width: 652px; }
        .container .fifteen.columns                 { width: 700px; }
        .container .sixteen.columns                 { width: 748px; }

        .container .one-third.column                { width: 236px; }
        .container .two-thirds.column               { width: 492px; }

        /* Offsets */
        .container .offset-by-one                   { padding-left: 48px; }
        .container .offset-by-two                   { padding-left: 96px; }
        .container .offset-by-three                 { padding-left: 144px; }
        .container .offset-by-four                  { padding-left: 192px; }
        .container .offset-by-five                  { padding-left: 240px; }
        .container .offset-by-six                   { padding-left: 288px; }
        .container .offset-by-seven                 { padding-left: 336px; }
        .container .offset-by-eight                 { padding-left: 384px; }
        .container .offset-by-nine                  { padding-left: 432px; }
        .container .offset-by-ten                   { padding-left: 480px; }
        .container .offset-by-eleven                { padding-left: 528px; }
        .container .offset-by-twelve                { padding-left: 576px; }
        .container .offset-by-thirteen              { padding-left: 624px; }
        .container .offset-by-fourteen              { padding-left: 672px; }
        .container .offset-by-fifteen               { padding-left: 720px; }
    }


/*  #Mobile (Portrait)
================================================== */

    /* Note: Design for a width of 320px */

    @media only screen and (max-width: 767px) {
        .container { width: 300px; }
        .container .columns,
        .container .column { margin: 0; }

        .container .one.column,
        .container .one.columns,
        .container .two.columns,
        .container .three.columns,
        .container .four.columns,
        .container .five.columns,
        .container .six.columns,
        .container .seven.columns,
        .container .eight.columns,
        .container .nine.columns,
        .container .ten.columns,
        .container .eleven.columns,
        .container .twelve.columns,
        .container .thirteen.columns,
        .container .fourteen.columns,
        .container .fifteen.columns,
        .container .sixteen.columns,
        .container .one-third.column,
        .container .two-thirds.column  { width: 300px; }

        /* Offsets */
        .container .offset-by-one,
        .container .offset-by-two,
        .container .offset-by-three,
        .container .offset-by-four,
        .container .offset-by-five,
        .container .offset-by-six,
        .container .offset-by-seven,
        .container .offset-by-eight,
        .container .offset-by-nine,
        .container .offset-by-ten,
        .container .offset-by-eleven,
        .container .offset-by-twelve,
        .container .offset-by-thirteen,
        .container .offset-by-fourteen,
        .container .offset-by-fifteen { padding-left: 0; }

    }


/* #Mobile (Landscape)
================================================== */

    /* Note: Design for a width of 480px */

    @media only screen and (min-width: 480px) and (max-width: 767px) {
        .container { width: 420px; }
        .container .columns,
        .container .column { margin: 0; }

        .container .one.column,
        .container .one.columns,
        .container .two.columns,
        .container .three.columns,
        .container .four.columns,
        .container .five.columns,
        .container .six.columns,
        .container .seven.columns,
        .container .eight.columns,
        .container .nine.columns,
        .container .ten.columns,
        .container .eleven.columns,
        .container .twelve.columns,
        .container .thirteen.columns,
        .container .fourteen.columns,
        .container .fifteen.columns,
        .container .sixteen.columns,
        .container .one-third.column,
        .container .two-thirds.column { width: 420px; }
    }


/* #Clearing
================================================== */

    /* Self Clearing Goodness */
    .container:after { content: "\0020"; display: block; height: 0; clear: both; visibility: hidden; }

    /* Use clearfix class on parent to clear nested columns,
    or wrap each row of columns in a <div class="row"> */
    .clearfix:before,
    .clearfix:after,
    .row:before,
    .row:after {
      content: '\0020';
      display: block;
      overflow: hidden;
      visibility: hidden;
      width: 0;
      height: 0; }
    .row:after,
    .clearfix:after {
      clear: both; }
    .row,
    .clearfix {
      zoom: 1; }

    /* You can also use a <br class="clear" /> to clear columns */
    .clear {
      clear: both;
      display: block;
      overflow: hidden;
      visibility: hidden;
      width: 0;
      height: 0;
    }