Fix reallocation of D buffer in gemm (2.4)

Cherry-picked #3106 onto 2.4