tabletserver: rewrite max dml rows test

The previous test was slapped together to make sure the
feature worked. This is a more robust version of the same
test.