The most common reason for batter to fall off is the oil is not hot enough. If it is below 350 degrees Fahrenheit, the fish will absorb some of the oil while frying. This causes it to become soggy and lose some of the batter.