I don't think Factory Girl hits the db, so that helps. Other than that I'd think avoiding over testing.

I've never worried too much about test db size tho. From what I've seen integration are usually  the slowest.