Sometimes, if fences block cold air drainage through your property, the cold air can puddle behind them causing very localized frost damage on near-freezing nights. Such impervious areas can't absorb water, and may create wet spots if the water that flows off of them is concentrated in one area. Watch for similar wet areas where water flows off roofs or out of downspouts. Trees can also prevent rain from reaching the ground. That, coupled with competition for water and nutrients from the tree's roots, may make it difficult to grow less-competitive plants around the base of larger trees.
Soil types can also affect frost.
Heavy clay soils can act much like paved surfaces, moderating the temperature near ground level. Lighter soils that have many air pockets in them can act as an insulating layer on top of warmer subsoils, trapping that heat below ground and are hence more prone to frosts at ground level. Contact your county's Cornell Cooperative Extension staff for local advice on microclimates.