Not all growers are prejudiced. Many have learned that attractive, clean, well maintained camps can attract workers who are careful and will care for the housing. Others still argue that it is useless to have clean housing because migrants are dirty and will ruin it. Others argue that the trend toward automation will make it a waste to fix up housing that will be useless in a few years. These farmers look forward to getting rid of their migrants.
Racism is certainly a factor in many cases.
Finally, fear of retaliation by powerful growers is a factor that keeps housing substandard. Negative action may cost farmworkers their job and whatever housing is available. Also it is costly to go to court even though it does work. Further scams are put over on the workers. For example, to avoid a suit over poor conditions, an owner signed over the housing to the worker for the season.