Fix a bug that response logger gets dead lock