Fix reallocation of D buffer in gemm

Cherry-picked #3106 onto 2.4