Fix bug where we were multiply device allocations by elem_size twice