Add min_heap_size to worker
Revert "Bump min_heap_size of bertconf_bert_loader"

This reverts commit b334e2ab8786629e3ea6cfbd902637608ed4a5d4.