Fix up asserts

We were using asserts in some places where an exception is probably more appropriate. Also found some missing checks for memory allocation, and pointless includes for assert.h