Performance Optimizations and Bug Fixes

This is a cleaned up version of #927 and #929 and includes all the changes of these two pull requests and also fixes a number of issues that were the result of a bad merge.