Revert "[MAT-450] Don't test mem manager"

This reverts commit 32b9ee93d90980adc3cfad9fb05362d775bad36d.
[MAT-450] Only build memory manager on 64 bit