add improved memory management in hadoop layer