I wasn't freeing the collision count array on the device. This may have been causing some of the memory problems.