The two main factors are the object's orbit and its size. If the trajectory of the object has no chance of intersecting Earth, then it's not a threat. If the object's path possibly does impact, we need to consider its size. Small impactors of a few meters in size or less will disintegrate in the atmosphere and cause no damage to the ground. Objects of tens of meters in size could cause localized damage, 100 meters or more regional scale damage, and 1 km or more global damage. The good news is that the expected frequency of impacts on Earth falls off dramatically for larger and larger objects. For example, objects larger than 1 km impact Earth no more frequently than every several hundred thousand years, on average. Impact events like the Chelyabinsk meteor of 2013, which was only about 20 m in size, happen on timescales of once every several decades, on average.