Yes, it is very bad to drive like this. The main reason for this is that these particles prevent the fluid from flowing correctly. So, they can increase the wear and tear in the transmission. This is why it is recommended that you flush the fluid if you notice that it’s dirty.