Minor tweak to how the WorkloadSummarizer calculates parameter signatures. Not much we can do to improve TPC-C.

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2328 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Added support back in for optional IDs fields in JSON traces

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2329 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Tweaked MostPopularPartitioner in order to make sure our designs don't run out of memory. Fixed VerticalPartition selections for multi-partition attributes

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2330 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Minor fixes for paths and checkpoints

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2331 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Ok now this is a good one:
+ The first part of TimeIntervalCostModel.estimateCost() is now multi-threaded using our simple Producer/Consumer classes
+ Removed DesignerUtil and move the important methods into either CatalogUtil or PartitionerUtil. Doesn't really make sense to call it 'DesignerUtil' when the runtime stuff needs it. This allows to take advantage of the caching component of CatalogUtil that's already in place.
+ Added additional checks to make sure that the MaterializedViewInfo's destination table is in the catalog


git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2332 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Removed some debugging messages

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2333 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Minor fixes for calculating the lower bounds of designs. These numbers seem too low. Need to investigate

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2334 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Worker on Producer/Consumer util. The WorkloadSummarizer is now multi-threaded because of it

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2335 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Finally fixed support for weighted workloads in TimeIntervalCostModel. It was a suprisingly easy fix. Add support for scaling database sizes using the TableStatisticsGenerators.

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2336 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Minor tweaks. Refactor the Fabric/Boto EC2 deployment script so that we can set the client and site instance types separately. Completely untested...

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2337 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Working on updated fabric script

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2338 7a8f3d5b-8c5f-44cb-9125-2414a256df87
This is a good one, so bear with me here...

We have a new HStoreThreadManager utility that we use to register EE and non-EE threads and then set their CPU core affinity.
Added new HStoreConf options site.cpu_affinity and site.cpu_affinity_one_partition_per_core that we can use to play around with different configurations on EC2 and understand what works the best. This provides a most speed boost per HStoreSite on EC2

Which leads me to now discover at the end of the day that what we really need is a separate incoming throttle per partition, not per HStoreSite. I've noticed some funkiness where one core will run slower than others on EC2. That might be because we keep queuing up incoming txns and have to acquire a lock in the ExecutionSite. Regardless this causes the HStoreSite to start shedding work, leaving some EE idle. We should really get rid of the single throttle and the ExecutionSite locks...
So that means I'm going to go home, eat dinner, and then bang out this fix.



git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2339 7a8f3d5b-8c5f-44cb-9125-2414a256df87
There is only an incoming txn queue limit, and it is now per partition. Haven't tried it on EC2 yet

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2340 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Working on scaling up single-node performance. Added a bunch of different profiling metrics. Still not sure why the queues randomly get empty after a while...

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2341 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Quick hackjob to add BatchPlanner profiling. I think the main thing I need to optimize is to add caching to the PartitionEstimator for single-partition batches...

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2342 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Significant Optimization: We can now cache single-partition BatchPlans. This prevents us from having to hit the PartitionEstimator for every query invocation. In some experiments using only NewOrder, this reduces the BatchPlanner overhead by +60%. 
The PartitionEstimator now uses CatalogKey less to store its internal data structures. This reduces the number of look-ups that we have to do. There are still other optimizations that could be done but I think that the BatchPlanner caching is going to be the major win.


git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2343 7a8f3d5b-8c5f-44cb-9125-2414a256df87
BatchPlanner.BatchPlan now longer keeps a cached list of FragmentTaskMessages. Again, optimizing for the common single-partition case

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2344 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Minor fixes for the SIGMOD partitioning experiment script

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2345 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Fixed a bug in HStoreSite where it would throttle VoltListeners based on the total # of inflight txns, rather than based on whether the specific ExecutionSite was overloaded.
More tweaks in the Fabric deployment script


git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2346 7a8f3d5b-8c5f-44cb-9125-2414a256df87
Minor tweak to fix failing test case

git-svn-id: https://database.cs.brown.edu/svn/hstore/branches/partitioning-branch@2347 7a8f3d5b-8c5f-44cb-9125-2414a256df87