Handle the mem allocations properly