CatalogMetricsStorage
1) Stores metrics in Local Catalog or Catalog in Azure
2) Every StatisticsCatalogItem has 1000 download stats in them. Currently,
this one writes to and commits the Catalog for every StatisticsCatalogItem
This happens sequentially though, does not handle concurrency

TODO
1) Test Catalog in Azure version
2) Add better support for concurrency. Commit to Catalog should happen from
a single thread
1) Using the default NameValueCollection WebConfigurationManager.AppSettings
directly instead of passing around too many parameters
2) Tested it successfully using a Local Website and Console App
1) Handling concurrency by way of a ConcurrentQueue and queue of queues
In the previous commit,
Add handling of concurrency by way of a ConcurrentQueue and queue of queues

In this commit,
Tested concurrency and made a few changes
1) Through config, Metrics Service could write both to DB and Catalog
2) Tested writing to Catalog in Azure using Development Storage. Also,
Test Console application can uses this
1) Catalog storage should be assumed based on CatalogIndexUrl or CatalogStorageAccount
2) Changed the settings to IDictionary<string, string>
Deleting all content files brought in by Metadata Catalog package
Changed app settings to be a simple IDictionary<string, string>
Incorporated comments around some app settings
1. Added PackageStatsHandlerTests class with simple tests
for Metrics Service App setting
2. In order for the test project to reference the source, added the test
project to the main solution
(Also, added changes to CatalogMetricsStorage that were missed in the
previous commit)
1) Using Task.Run from EnqueueStats to commit to catalog instead of a
dedicated thread
2) Using an int called 'CatalogWriterGate' to determine if there is a
thread writing to Catalog. If the int is 0, the writer is open for writing
if it is 1, then it is closed for writing
3) Thoroughly tested the logic by setting the CatalogItemSizeCount to 2
on the ConsoleMetricsTestApp and making 500 posts to the metrics service.
Successfully tested that multiple threads check and skip from committing to
Catalog. Also, tested that all the items are successfully added to Catalog
Updated to latest package of NuGet.Services.Metadata.Catalog
Incorporated comments from the second iteration

1) Changed package download stats packet to be a key-value pair instead of
just a flat list of values
2) Also, added columns names to the select query against DB
3) Added a try-catch on the CommitCatalog to catch all exceptions and
always open the CatalogWriterGate
Implemented catalog generation for metrics service with multiple instances

1) Each instance of the metrics service website creates a catalog with a
fixed prefix of website_instance_id
2) Also, creates a file called indexes/<website_instance_id>.txt for the
catalog collector to use

P.S: Removed blob directory function and split that into 2
1) Changed indexes file to be a simple json with keys 'CatalogIndexPath'
and 'CatalogLastUpdated'
2) Using Storage of the root of all
catalog to created indexes file which is cleaner instead of an if and else
3) Also, changed CatalogIndexUrlKey to CatalogBaseAddressKey for clarity

Testing...
1) Added a CatalogFlushTimer whose time period is a configurable
int, CatalogFlushTimePeriod. Default time period is 1 min or 60000 ms

2) Moved swapping of CurrentStatsQueue inside CommitToCatalog method

3) Tested the code by posting 3000 stats to 2 instances of the service.
At the end of posting, Each instance had around 500 items in memory
since each catalog item holds a 1000. But, after a min (flush time period),
the items in memory were successfully flushed
1) Moved the swapping of CurrentStatsQueue back to EnqueueStats. Added 1
to the TimerTask method as well. Swapping needs to happen as part of the
check. Otherwise, based on incoming rate, the queue may get higher than 1000
2) Even within EnqueueStats method, moved the enqueue statement after the
check and swap for the reasons stated above
After deploying the changes to a temporary website with 2 instances and
setting up the configuration as desired, I have added more logging and
error handling
Merge pull request #15 from NuGet/daravind/CatalogMetricsStorage

Daravind/Catalog metrics storage